[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407783529717-22743-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:29:43 dist-test-slave-wdjc krb5kdc[22750](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:29:43 dist-test-slave-wdjc krb5kdc[22750](info): set up 2 sockets
Sep 26 07:29:43 dist-test-slave-wdjc krb5kdc[22750](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260926 07:29:44.625125 22743 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407783529717-22743-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407783529717-22743-0/krb5kdc/test-user.keytab.
Sep 26 07:29:44 dist-test-slave-wdjc krb5kdc[22750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407784, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:29:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:29:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.53.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.53.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.53.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab.
Entry for principal kudu/127.22.53.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab.
I20260926 07:29:44.756686 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407783529717-22743-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407783529717-22743-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407783529717-22743-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.53.254:34671
--webserver_interface=127.22.53.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab
--principal=kudu/127.22.53.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:41973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.53.254:34671
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:29:44.875362 22763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:29:44.875770 22763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:29:44.875855 22763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:29:44.880206 22763 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:29:44.880355 22763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:29:44.880389 22763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:29:44.880420 22763 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:29:44.880460 22763 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:29:44.886399 22763 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:41973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407783529717-22743-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407783529717-22743-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.53.254:34671
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab
--principal=kudu/127.22.53.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.53.254:34671
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.53.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22763
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407783529717-22743-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:29:44.887744 22763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:29:44.888931 22763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:29:44.898708 22768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:29:44.898654 22771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:29:44.898804 22763 server_base.cc:1034] running on GCE node
W20260926 07:29:44.898654 22769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:29:44.899761 22763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:29:44.901104 22763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:29:44.902315 22763 hybrid_clock.cc:648] HybridClock initialized: now 1790407784902289 us; error 45 us; skew 500 ppm
Sep 26 07:29:44 dist-test-slave-wdjc krb5kdc[22750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407784, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:29:44.905524 22763 init.cc:377] Logged in from keytab as kudu/127.22.53.254@KRBTEST.COM (short username kudu)
I20260926 07:29:44.906522 22763 webserver.cc:460] Webserver started at http://127.22.53.254:46133/ using document root <none> and password file <none>
I20260926 07:29:44.907104 22763 fs_manager.cc:359] Metadata directory not provided
I20260926 07:29:44.907169 22763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:29:44.907351 22763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:29:44.909695 22763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407783529717-22743-0/minicluster-data/master-0/data/instance:
uuid: "2865181bedff4978ae1b8c782c38ce68"
format_stamp: "Formatted at 2026-09-26 07:29:44 on dist-test-slave-wdjc"
server_key: "4cf7200cbe96680a7f07463c34fdb2d3"
server_key_iv: "0bd52b8e71549eef24b890c3b09e3ddf"
server_key_version: "encryptionkey@0"
I20260926 07:29:44.910283 22763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407783529717-22743-0/minicluster-data/master-0/wal/instance:
uuid: "2865181bedff4978ae1b8c782c38ce68"
format_stamp: "Formatted at 2026-09-26 07:29:44 on dist-test-slave-wdjc"
server_key: "4cf7200cbe96680a7f07463c34fdb2d3"
server_key_iv: "0bd52b8e71549eef24b890c3b09e3ddf"
server_key_version: "encryptionkey@0"
I20260926 07:29:44.914530 22763 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 07:29:44.917534 22778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:44.919044 22763 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 07:29:44.919226 22763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407783529717-22743-0/minicluster-data/master-0/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407783529717-22743-0/minicluster-data/master-0/wal
uuid: "2865181bedff4978ae1b8c782c38ce68"
format_stamp: "Formatted at 2026-09-26 07:29:44 on dist-test-slave-wdjc"
server_key: "4cf7200cbe96680a7f07463c34fdb2d3"
server_key_iv: "0bd52b8e71549eef24b890c3b09e3ddf"
server_key_version: "encryptionkey@0"
I20260926 07:29:44.919373 22763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407783529717-22743-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407783529717-22743-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407783529717-22743-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:29:44.934165 22763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:29:44.937813 22763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:29:44.938061 22763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:29:44.946060 22763 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.254:34671
I20260926 07:29:44.946086 22830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.254:34671 every 8 connection(s)
I20260926 07:29:44.947150 22763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
I20260926 07:29:44.950318 22831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:29:44.954331 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 22763
I20260926 07:29:44.954460 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407783529717-22743-0/minicluster-data/master-0/wal/instance
I20260926 07:29:44.955832 22743 external_mini_cluster.cc:1428] Setting key 66dd0a2694bc4220552d6c161ed798f9
I20260926 07:29:44.956369 22831 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2865181bedff4978ae1b8c782c38ce68: Bootstrap starting.
I20260926 07:29:44.959082 22831 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2865181bedff4978ae1b8c782c38ce68: Neither blocks nor log segments found. Creating new log.
I20260926 07:29:44.959882 22831 log.cc:824] T 00000000000000000000000000000000 P 2865181bedff4978ae1b8c782c38ce68: Log is configured to *not* fsync() on all Append() calls
I20260926 07:29:44.961884 22831 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2865181bedff4978ae1b8c782c38ce68: No bootstrap required, opened a new log
I20260926 07:29:44.965919 22831 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2865181bedff4978ae1b8c782c38ce68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2865181bedff4978ae1b8c782c38ce68" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 34671 } }
I20260926 07:29:44.966223 22831 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2865181bedff4978ae1b8c782c38ce68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:29:44.966280 22831 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2865181bedff4978ae1b8c782c38ce68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2865181bedff4978ae1b8c782c38ce68, State: Initialized, Role: FOLLOWER
I20260926 07:29:44.966809 22831 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2865181bedff4978ae1b8c782c38ce68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2865181bedff4978ae1b8c782c38ce68" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 34671 } }
I20260926 07:29:44.966981 22831 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2865181bedff4978ae1b8c782c38ce68 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:29:44.967072 22831 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2865181bedff4978ae1b8c782c38ce68 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:29:44.967198 22831 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2865181bedff4978ae1b8c782c38ce68 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:29:44.968331 22831 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2865181bedff4978ae1b8c782c38ce68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2865181bedff4978ae1b8c782c38ce68" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 34671 } }
I20260926 07:29:44.968896 22831 leader_election.cc:302] T 00000000000000000000000000000000 P 2865181bedff4978ae1b8c782c38ce68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2865181bedff4978ae1b8c782c38ce68; no voters:
I20260926 07:29:44.969282 22831 leader_election.cc:288] T 00000000000000000000000000000000 P 2865181bedff4978ae1b8c782c38ce68 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:29:44.969399 22836 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2865181bedff4978ae1b8c782c38ce68 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:29:44.969655 22836 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2865181bedff4978ae1b8c782c38ce68 [term 1 LEADER]: Becoming Leader. State: Replica: 2865181bedff4978ae1b8c782c38ce68, State: Running, Role: LEADER
I20260926 07:29:44.969986 22836 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2865181bedff4978ae1b8c782c38ce68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2865181bedff4978ae1b8c782c38ce68" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 34671 } }
I20260926 07:29:44.970712 22831 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2865181bedff4978ae1b8c782c38ce68 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:29:44.971827 22838 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2865181bedff4978ae1b8c782c38ce68 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2865181bedff4978ae1b8c782c38ce68. Latest consensus state: current_term: 1 leader_uuid: "2865181bedff4978ae1b8c782c38ce68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2865181bedff4978ae1b8c782c38ce68" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 34671 } } }
I20260926 07:29:44.971983 22838 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2865181bedff4978ae1b8c782c38ce68 [sys.catalog]: This master's current role is: LEADER
I20260926 07:29:44.972373 22837 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2865181bedff4978ae1b8c782c38ce68 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2865181bedff4978ae1b8c782c38ce68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2865181bedff4978ae1b8c782c38ce68" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 34671 } } }
I20260926 07:29:44.972473 22837 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2865181bedff4978ae1b8c782c38ce68 [sys.catalog]: This master's current role is: LEADER
I20260926 07:29:44.972818 22845 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:29:44.977068 22845 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 26 07:29:44 dist-test-slave-wdjc krb5kdc[22750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407784, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.53.254@KRBTEST.COM
I20260926 07:29:44.984838 22845 catalog_manager.cc:1347] Generated new cluster ID: d17d45494d3f4f5f987edd13e5a4e91e
I20260926 07:29:44.984920 22845 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:29:44.985284 22833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:44.958928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40886 (local address 127.22.53.254:34671)
0926 07:29:44.959466 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:44.959489 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:44.960673 (+ 1184us) server_negotiation.cc:407] Connection header received
0926 07:29:44.965515 (+ 4842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:44.965539 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:44.965927 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:44.966349 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:44.969373 (+ 3024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:44.970837 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:44.972895 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:44.973677 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:44.973860 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:44.978411 (+ 4551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:44.978417 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:44.978431 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:29:44.978468 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:44.980714 (+ 2246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:44.981542 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:44.981549 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:44.981554 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:44.981701 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:44.982118 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:44.982121 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:44.982123 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:44.982656 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:44.982890 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:44.983656 (+ 766us) server_negotiation.cc:299] Negotiation successful
0926 07:29:44.983979 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":146,"threads_started":1}
I20260926 07:29:44.994946 22845 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:29:44.996004 22845 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:29:45.004837 22845 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2865181bedff4978ae1b8c782c38ce68: Generated new TSK 0
I20260926 07:29:45.005846 22845 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.53.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.53.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.53.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.193.keytab.
Entry for principal kudu/127.22.53.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.193.keytab.
I20260926 07:29:45.040076 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407783529717-22743-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407783529717-22743-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407783529717-22743-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407783529717-22743-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.53.193:0
--local_ip_for_outbound_sockets=127.22.53.193
--webserver_interface=127.22.53.193
--webserver_port=0
--tserver_master_addrs=127.22.53.254:34671
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.193.keytab
--principal=kudu/127.22.53.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:41973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:29:45.167778 22857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:29:45.168185 22857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:29:45.168237 22857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:29:45.173002 22857 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:29:45.173116 22857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:29:45.173188 22857 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.53.193
I20260926 07:29:45.179198 22857 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:41973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407783529717-22743-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407783529717-22743-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.193.keytab
--principal=kudu/127.22.53.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.53.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407783529717-22743-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.53.193
--webserver_port=0
--tserver_master_addrs=127.22.53.254:34671
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22857
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.53.193
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407783529717-22743-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:29:45.180390 22857 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:29:45.181541 22857 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:29:45.192198 22862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:29:45.192339 22865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:29:45.192198 22863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:29:45.192927 22857 server_base.cc:1034] running on GCE node
I20260926 07:29:45.193492 22857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:29:45.194214 22857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:29:45.195413 22857 hybrid_clock.cc:648] HybridClock initialized: now 1790407785195392 us; error 41 us; skew 500 ppm
Sep 26 07:29:45 dist-test-slave-wdjc krb5kdc[22750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407785, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:29:45.198755 22857 init.cc:377] Logged in from keytab as kudu/127.22.53.193@KRBTEST.COM (short username kudu)
I20260926 07:29:45.199811 22857 webserver.cc:460] Webserver started at http://127.22.53.193:40419/ using document root <none> and password file <none>
I20260926 07:29:45.200425 22857 fs_manager.cc:359] Metadata directory not provided
I20260926 07:29:45.200493 22857 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:29:45.200776 22857 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:29:45.203104 22857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407783529717-22743-0/minicluster-data/ts-0/data/instance:
uuid: "72a8ab1c4aff41fda2d7c36dbeac38d2"
format_stamp: "Formatted at 2026-09-26 07:29:45 on dist-test-slave-wdjc"
server_key: "5265de6b0b9e2f91195ac7135a998974"
server_key_iv: "ee913869f29b6cc0c7d4232db7c5032d"
server_key_version: "encryptionkey@0"
I20260926 07:29:45.203732 22857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407783529717-22743-0/minicluster-data/ts-0/wal/instance:
uuid: "72a8ab1c4aff41fda2d7c36dbeac38d2"
format_stamp: "Formatted at 2026-09-26 07:29:45 on dist-test-slave-wdjc"
server_key: "5265de6b0b9e2f91195ac7135a998974"
server_key_iv: "ee913869f29b6cc0c7d4232db7c5032d"
server_key_version: "encryptionkey@0"
I20260926 07:29:45.208235 22857 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 07:29:45.211457 22872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:45.213176 22857 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:29:45.213348 22857 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407783529717-22743-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407783529717-22743-0/minicluster-data/ts-0/wal
uuid: "72a8ab1c4aff41fda2d7c36dbeac38d2"
format_stamp: "Formatted at 2026-09-26 07:29:45 on dist-test-slave-wdjc"
server_key: "5265de6b0b9e2f91195ac7135a998974"
server_key_iv: "ee913869f29b6cc0c7d4232db7c5032d"
server_key_version: "encryptionkey@0"
I20260926 07:29:45.213507 22857 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407783529717-22743-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407783529717-22743-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407783529717-22743-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:29:45.231614 22857 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:29:45.235867 22857 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:29:45.236056 22857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:29:45.236732 22857 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:29:45.237612 22857 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:29:45.237663 22857 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:45.237703 22857 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:29:45.237720 22857 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:45.249042 22857 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.193:46749
I20260926 07:29:45.249056 22985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.193:46749 every 8 connection(s)
I20260926 07:29:45.250247 22857 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407783529717-22743-0/minicluster-data/ts-0/data/info.pb
I20260926 07:29:45.258116 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 22857
I20260926 07:29:45.258291 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407783529717-22743-0/minicluster-data/ts-0/wal/instance
I20260926 07:29:45.258827 22743 external_mini_cluster.cc:1428] Setting key 784ff44121b405bb3370ed3970b3a35e
Sep 26 07:29:45 dist-test-slave-wdjc krb5kdc[22750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407785, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.193@KRBTEST.COM for kudu/127.22.53.254@KRBTEST.COM
I20260926 07:29:45.269666 22833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:45.252697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.193:60603 (local address 127.22.53.254:34671)
0926 07:29:45.252895 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:45.252900 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:45.254398 (+ 1498us) server_negotiation.cc:407] Connection header received
0926 07:29:45.255459 (+ 1061us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:45.255464 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:45.255533 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:45.255645 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:45.257284 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:45.257895 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:45.260019 (+ 2124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:45.260631 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:45.260812 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:45.264621 (+ 3809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:45.264629 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:45.264635 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:29:45.264680 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:45.266727 (+ 2047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:45.267417 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:45.267424 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:45.267428 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:45.267549 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:45.267989 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:45.267996 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:45.267999 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:45.268332 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:45.268477 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:45.269196 (+ 719us) server_negotiation.cc:299] Negotiation successful
0926 07:29:45.269395 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":49,"server-negotiator.queue_time_us":69}
I20260926 07:29:45.270469 22988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:45.253243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:34671 (local address 127.22.53.193:60603)
0926 07:29:45.254120 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:45.254174 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 07:29:45.255119 (+ 945us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:29:45.255907 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:29:45.255916 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:29:45.256376 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:29:45.257034 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:45.257052 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:45.258101 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:45.258106 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:45.259769 (+ 1663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:45.259789 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:45.260871 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:45.260875 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:45.261146 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:45.262009 (+ 863us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:29:45.262038 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:29:45.264315 (+ 2277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:29:45.266919 (+ 2604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:29:45.266924 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:29:45.266942 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:29:45.267284 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:29:45.267680 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:29:45.267683 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:29:45.267684 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:29:45.267848 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:29:45.268499 (+ 651us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:29:45.268505 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:29:45.268894 (+ 389us) client_negotiation.cc:769] Sending connection context
0926 07:29:45.269103 (+ 209us) client_negotiation.cc:240] Negotiation successful
0926 07:29:45.269414 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"thread_start_us":161,"threads_started":1}
WARNING: no policy specified for kudu/127.22.53.194@KRBTEST.COM; defaulting to no policy
I20260926 07:29:45.272456 22986 heartbeater.cc:347] Connected to a master server at 127.22.53.254:34671
I20260926 07:29:45.272796 22986 heartbeater.cc:464] Registering TS with master...
I20260926 07:29:45.273458 22986 heartbeater.cc:511] Master 127.22.53.254:34671 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.53.194@KRBTEST.COM" created.
I20260926 07:29:45.275262 22795 ts_manager.cc:194] Registered new tserver with Master: 72a8ab1c4aff41fda2d7c36dbeac38d2 (127.22.53.193:46749)
I20260926 07:29:45.277099 22795 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.53.193@KRBTEST.COM'} at 127.22.53.193:60603
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.53.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.194.keytab.
Entry for principal kudu/127.22.53.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.194.keytab.
I20260926 07:29:45.292361 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407783529717-22743-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407783529717-22743-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407783529717-22743-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407783529717-22743-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.53.194:0
--local_ip_for_outbound_sockets=127.22.53.194
--webserver_interface=127.22.53.194
--webserver_port=0
--tserver_master_addrs=127.22.53.254:34671
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.194.keytab
--principal=kudu/127.22.53.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:41973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:29:45.415787 22991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:29:45.416100 22991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:29:45.416149 22991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:29:45.421236 22991 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:29:45.421331 22991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:29:45.421401 22991 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.53.194
I20260926 07:29:45.427479 22991 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:41973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407783529717-22743-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407783529717-22743-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.194.keytab
--principal=kudu/127.22.53.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.53.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407783529717-22743-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.53.194
--webserver_port=0
--tserver_master_addrs=127.22.53.254:34671
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22991
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.53.194
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407783529717-22743-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:29:45.428814 22991 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:29:45.429963 22991 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:29:45.441041 22999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:29:45.441051 22996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:29:45.441093 22991 server_base.cc:1034] running on GCE node
W20260926 07:29:45.441040 22997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:29:45.442025 22991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:29:45.442826 22991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:29:45.444017 22991 hybrid_clock.cc:648] HybridClock initialized: now 1790407785443981 us; error 57 us; skew 500 ppm
Sep 26 07:29:45 dist-test-slave-wdjc krb5kdc[22750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407785, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:29:45.447471 22991 init.cc:377] Logged in from keytab as kudu/127.22.53.194@KRBTEST.COM (short username kudu)
I20260926 07:29:45.448649 22991 webserver.cc:460] Webserver started at http://127.22.53.194:35851/ using document root <none> and password file <none>
I20260926 07:29:45.449326 22991 fs_manager.cc:359] Metadata directory not provided
I20260926 07:29:45.449402 22991 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:29:45.449656 22991 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:29:45.452028 22991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407783529717-22743-0/minicluster-data/ts-1/data/instance:
uuid: "a9a55a95371a447698fae3ad6bde944d"
format_stamp: "Formatted at 2026-09-26 07:29:45 on dist-test-slave-wdjc"
server_key: "2df8c80b418548db41212a879719c95a"
server_key_iv: "dc32189d2a06914f4cdef2b5c97997fb"
server_key_version: "encryptionkey@0"
I20260926 07:29:45.452765 22991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407783529717-22743-0/minicluster-data/ts-1/wal/instance:
uuid: "a9a55a95371a447698fae3ad6bde944d"
format_stamp: "Formatted at 2026-09-26 07:29:45 on dist-test-slave-wdjc"
server_key: "2df8c80b418548db41212a879719c95a"
server_key_iv: "dc32189d2a06914f4cdef2b5c97997fb"
server_key_version: "encryptionkey@0"
I20260926 07:29:45.457494 22991 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 07:29:45.460464 23006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:45.462203 22991 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 07:29:45.462399 22991 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407783529717-22743-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407783529717-22743-0/minicluster-data/ts-1/wal
uuid: "a9a55a95371a447698fae3ad6bde944d"
format_stamp: "Formatted at 2026-09-26 07:29:45 on dist-test-slave-wdjc"
server_key: "2df8c80b418548db41212a879719c95a"
server_key_iv: "dc32189d2a06914f4cdef2b5c97997fb"
server_key_version: "encryptionkey@0"
I20260926 07:29:45.462584 22991 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407783529717-22743-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407783529717-22743-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407783529717-22743-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:29:45.480211 22991 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:29:45.484411 22991 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:29:45.484800 22991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:29:45.485512 22991 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:29:45.486503 22991 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:29:45.486586 22991 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:45.486671 22991 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:29:45.486723 22991 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:45.497292 22991 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.194:35345
I20260926 07:29:45.497313 23119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.194:35345 every 8 connection(s)
I20260926 07:29:45.498306 22991 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407783529717-22743-0/minicluster-data/ts-1/data/info.pb
I20260926 07:29:45.500344 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 22991
I20260926 07:29:45.500495 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407783529717-22743-0/minicluster-data/ts-1/wal/instance
I20260926 07:29:45.500887 22743 external_mini_cluster.cc:1428] Setting key 07d2e2216baf62f16b0b00adbd33e370
Sep 26 07:29:45 dist-test-slave-wdjc krb5kdc[22750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407785, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.194@KRBTEST.COM for kudu/127.22.53.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.22.53.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.53.195@KRBTEST.COM" created.
I20260926 07:29:45.517232 22833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:45.500475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.194:44021 (local address 127.22.53.254:34671)
0926 07:29:45.500688 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:45.500695 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:45.501986 (+ 1291us) server_negotiation.cc:407] Connection header received
0926 07:29:45.503108 (+ 1122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:45.503111 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:45.503180 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:45.503256 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:45.505195 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:45.506194 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:45.508415 (+ 2221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:45.508771 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:45.508875 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:45.512600 (+ 3725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:45.512613 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:45.512617 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:29:45.512669 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:45.514498 (+ 1829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:45.515197 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:45.515202 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:45.515205 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:45.515312 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:45.515788 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:45.515794 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:45.515796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:45.516027 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:45.516161 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:45.516868 (+ 707us) server_negotiation.cc:299] Negotiation successful
0926 07:29:45.517027 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260926 07:29:45.518314 23122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:45.501003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:34671 (local address 127.22.53.194:44021)
0926 07:29:45.501799 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:45.501859 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0926 07:29:45.502811 (+ 952us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:29:45.503523 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:29:45.503537 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:29:45.504197 (+ 660us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:29:45.505012 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:45.505031 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:45.506412 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:45.506419 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:45.508241 (+ 1822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:45.508262 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:45.508994 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:45.509002 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:45.509301 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:45.510190 (+ 889us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:29:45.510216 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:29:45.512286 (+ 2070us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:29:45.514675 (+ 2389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:29:45.514683 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:29:45.514706 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:29:45.515040 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:29:45.515432 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:29:45.515436 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:29:45.515438 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:29:45.515675 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:29:45.516161 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:29:45.516170 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:29:45.516505 (+ 335us) client_negotiation.cc:769] Sending connection context
0926 07:29:45.516819 (+ 314us) client_negotiation.cc:240] Negotiation successful
0926 07:29:45.517253 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":150,"threads_started":1}
I20260926 07:29:45.519927 23120 heartbeater.cc:347] Connected to a master server at 127.22.53.254:34671
I20260926 07:29:45.520524 23120 heartbeater.cc:464] Registering TS with master...
I20260926 07:29:45.521206 23120 heartbeater.cc:511] Master 127.22.53.254:34671 requested a full tablet report, sending...
I20260926 07:29:45.522589 22795 ts_manager.cc:194] Registered new tserver with Master: a9a55a95371a447698fae3ad6bde944d (127.22.53.194:35345)
I20260926 07:29:45.523712 22795 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.53.194@KRBTEST.COM'} at 127.22.53.194:44021
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.53.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.195.keytab.
Entry for principal kudu/127.22.53.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.195.keytab.
I20260926 07:29:45.536192 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407783529717-22743-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407783529717-22743-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407783529717-22743-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407783529717-22743-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.53.195:0
--local_ip_for_outbound_sockets=127.22.53.195
--webserver_interface=127.22.53.195
--webserver_port=0
--tserver_master_addrs=127.22.53.254:34671
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.195.keytab
--principal=kudu/127.22.53.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:41973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:29:45.660246 23125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:29:45.660713 23125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:29:45.660774 23125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:29:45.665498 23125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:29:45.665584 23125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:29:45.665653 23125 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.53.195
I20260926 07:29:45.671933 23125 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:41973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407783529717-22743-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407783529717-22743-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.195.keytab
--principal=kudu/127.22.53.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.53.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407783529717-22743-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.53.195
--webserver_port=0
--tserver_master_addrs=127.22.53.254:34671
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23125
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.53.195
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407783529717-22743-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:29:45.673398 23125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:29:45.674525 23125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:29:45.685413 23130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:29:45.685415 23133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:29:45.685415 23131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:29:45.685578 23125 server_base.cc:1034] running on GCE node
I20260926 07:29:45.686368 23125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:29:45.687173 23125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:29:45.688381 23125 hybrid_clock.cc:648] HybridClock initialized: now 1790407785688364 us; error 38 us; skew 500 ppm
Sep 26 07:29:45 dist-test-slave-wdjc krb5kdc[22750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407785, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:29:45.694883 23125 init.cc:377] Logged in from keytab as kudu/127.22.53.195@KRBTEST.COM (short username kudu)
I20260926 07:29:45.696204 23125 webserver.cc:460] Webserver started at http://127.22.53.195:33061/ using document root <none> and password file <none>
I20260926 07:29:45.696959 23125 fs_manager.cc:359] Metadata directory not provided
I20260926 07:29:45.697072 23125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:29:45.697346 23125 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:29:45.699752 23125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407783529717-22743-0/minicluster-data/ts-2/data/instance:
uuid: "d7f502ea30b0430ea86adf894453921a"
format_stamp: "Formatted at 2026-09-26 07:29:45 on dist-test-slave-wdjc"
server_key: "2a795f7929861027115f5b61393c887c"
server_key_iv: "d3ee74f1261eda00a2835c5169f0cacf"
server_key_version: "encryptionkey@0"
I20260926 07:29:45.700487 23125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407783529717-22743-0/minicluster-data/ts-2/wal/instance:
uuid: "d7f502ea30b0430ea86adf894453921a"
format_stamp: "Formatted at 2026-09-26 07:29:45 on dist-test-slave-wdjc"
server_key: "2a795f7929861027115f5b61393c887c"
server_key_iv: "d3ee74f1261eda00a2835c5169f0cacf"
server_key_version: "encryptionkey@0"
I20260926 07:29:45.705088 23125 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 07:29:45.708079 23140 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:45.709610 23125 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:29:45.709820 23125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407783529717-22743-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407783529717-22743-0/minicluster-data/ts-2/wal
uuid: "d7f502ea30b0430ea86adf894453921a"
format_stamp: "Formatted at 2026-09-26 07:29:45 on dist-test-slave-wdjc"
server_key: "2a795f7929861027115f5b61393c887c"
server_key_iv: "d3ee74f1261eda00a2835c5169f0cacf"
server_key_version: "encryptionkey@0"
I20260926 07:29:45.709975 23125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407783529717-22743-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407783529717-22743-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407783529717-22743-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:29:45.743224 23125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:29:45.746886 23125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:29:45.747164 23125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:29:45.747820 23125 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:29:45.748777 23125 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:29:45.748859 23125 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:45.748934 23125 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:29:45.748976 23125 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:45.759166 23125 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.195:43691
I20260926 07:29:45.759192 23253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.195:43691 every 8 connection(s)
I20260926 07:29:45.760239 23125 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407783529717-22743-0/minicluster-data/ts-2/data/info.pb
I20260926 07:29:45.764714 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 23125
I20260926 07:29:45.764853 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407783529717-22743-0/minicluster-data/ts-2/wal/instance
I20260926 07:29:45.765177 22743 external_mini_cluster.cc:1428] Setting key 0053755303ac3a0d3b75714b1316a256
Sep 26 07:29:45 dist-test-slave-wdjc krb5kdc[22750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407785, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.195@KRBTEST.COM for kudu/127.22.53.254@KRBTEST.COM
I20260926 07:29:45.782883 22833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:45.762338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.195:53421 (local address 127.22.53.254:34671)
0926 07:29:45.762487 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:45.762491 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:45.763886 (+ 1395us) server_negotiation.cc:407] Connection header received
0926 07:29:45.765493 (+ 1607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:45.765497 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:45.765591 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:45.768148 (+ 2557us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:45.770324 (+ 2176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:45.771066 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:45.773131 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:45.773496 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:45.773633 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:45.777523 (+ 3890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:45.777527 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:45.777529 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:29:45.777563 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:45.779546 (+ 1983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:45.780603 (+ 1057us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:45.780609 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:45.780612 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:45.780742 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:45.781317 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:45.781322 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:45.781326 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:45.781651 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:45.781779 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:45.782602 (+ 823us) server_negotiation.cc:299] Negotiation successful
0926 07:29:45.782747 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 07:29:45.784240 23256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:45.762734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:34671 (local address 127.22.53.195:53421)
0926 07:29:45.763682 (+ 948us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:45.763760 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0926 07:29:45.765163 (+ 1403us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:29:45.768450 (+ 3287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:29:45.768466 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:29:45.769140 (+ 674us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:29:45.770083 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:45.770127 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:45.771219 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:45.771223 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:45.772964 (+ 1741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:45.772978 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:45.773755 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:45.773760 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:45.774049 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:45.775162 (+ 1113us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:29:45.775186 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:29:45.777252 (+ 2066us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:29:45.779767 (+ 2515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:29:45.779775 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:29:45.779805 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:29:45.780386 (+ 581us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:29:45.780889 (+ 503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:29:45.780896 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:29:45.780899 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:29:45.781164 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:29:45.781811 (+ 647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:29:45.781822 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:29:45.782254 (+ 432us) client_negotiation.cc:769] Sending connection context
0926 07:29:45.782510 (+ 256us) client_negotiation.cc:240] Negotiation successful
0926 07:29:45.782939 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":657,"thread_start_us":133,"threads_started":1}
I20260926 07:29:45.786013 23254 heartbeater.cc:347] Connected to a master server at 127.22.53.254:34671
I20260926 07:29:45.788199 23254 heartbeater.cc:464] Registering TS with master...
I20260926 07:29:45.789103 23254 heartbeater.cc:511] Master 127.22.53.254:34671 requested a full tablet report, sending...
I20260926 07:29:45.790645 22795 ts_manager.cc:194] Registered new tserver with Master: d7f502ea30b0430ea86adf894453921a (127.22.53.195:43691)
I20260926 07:29:45.791636 22795 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.53.195@KRBTEST.COM'} at 127.22.53.195:53421
I20260926 07:29:45.797748 22743 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 07:29:45 dist-test-slave-wdjc krb5kdc[22750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407785, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 07:29:45 dist-test-slave-wdjc krb5kdc[22750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407785, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.22.53.193@KRBTEST.COM
I20260926 07:29:45.833214 23261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:45.821694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55860 (local address 127.22.53.193:46749)
0926 07:29:45.822106 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:45.822113 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:45.822139 (+ 26us) server_negotiation.cc:407] Connection header received
0926 07:29:45.822315 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:45.822322 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:45.822536 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:45.822669 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:45.823661 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:45.824660 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:45.825588 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:45.826043 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:45.826149 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:45.829081 (+ 2932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:45.829106 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:45.829124 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:29:45.829161 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:45.831038 (+ 1877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:45.831603 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:45.831610 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:45.831612 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:45.831730 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:45.832006 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:45.832009 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:45.832010 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:45.832291 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:45.832468 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:45.832904 (+ 436us) server_negotiation.cc:299] Negotiation successful
0926 07:29:45.833065 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":157,"threads_started":1}
W20260926 07:29:45.834479 22900 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55860
Sep 26 07:29:45 dist-test-slave-wdjc krb5kdc[22750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407785, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 07:29:45 dist-test-slave-wdjc krb5kdc[22750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407785, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.53.193@KRBTEST.COM
I20260926 07:29:45.863407 23261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:45.854092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55866 (local address 127.22.53.193:46749)
0926 07:29:45.854306 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:45.854311 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:45.854367 (+ 56us) server_negotiation.cc:407] Connection header received
0926 07:29:45.854585 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:45.854589 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:45.854654 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:45.854752 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:45.855570 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:45.856145 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:45.857157 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:45.857412 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:45.857504 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:45.860120 (+ 2616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:45.860124 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:45.860127 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:29:45.860159 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:45.861650 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:45.862126 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:45.862129 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:45.862131 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:45.862212 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:45.862511 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:45.862514 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:45.862515 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:45.862713 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:45.862817 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:45.863081 (+ 264us) server_negotiation.cc:299] Negotiation successful
0926 07:29:45.863194 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260926 07:29:45.864722 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 22857
I20260926 07:29:45.871455 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 22991
I20260926 07:29:45.877274 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 23125
I20260926 07:29:45.883093 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 22763
2026-09-26T07:29:45Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2347 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407783529717-22743-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:29:45 dist-test-slave-wdjc krb5kdc[23268](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:29:45 dist-test-slave-wdjc krb5kdc[23268](info): set up 2 sockets
Sep 26 07:29:45 dist-test-slave-wdjc krb5kdc[23268](info): commencing operation
krb5kdc: starting...
W20260926 07:29:46.969041 22743 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407783529717-22743-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407783529717-22743-0/krb5kdc/test-user.keytab.
Sep 26 07:29:47 dist-test-slave-wdjc krb5kdc[23268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407787, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:29:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:29:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.53.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.53.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.53.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab.
Entry for principal kudu/127.22.53.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab.
I20260926 07:29:47.099912 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407783529717-22743-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407783529717-22743-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407783529717-22743-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.53.254:42721
--webserver_interface=127.22.53.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab
--principal=kudu/127.22.53.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:45713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.53.254:42721
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:29:47.221505 23281 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:29:47.221879 23281 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:29:47.221967 23281 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:29:47.226686 23281 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:29:47.226820 23281 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:29:47.226861 23281 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:29:47.226922 23281 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:29:47.226958 23281 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:29:47.233172 23281 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:45713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407783529717-22743-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407783529717-22743-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.53.254:42721
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab
--principal=kudu/127.22.53.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.53.254:42721
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.53.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23281
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407783529717-22743-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:29:47.234437 23281 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:29:47.235574 23281 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:29:47.245020 23287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:29:47.245020 23286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:29:47.245208 23281 server_base.cc:1034] running on GCE node
W20260926 07:29:47.245456 23289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:29:47.246125 23281 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:29:47.247437 23281 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:29:47.248636 23281 hybrid_clock.cc:648] HybridClock initialized: now 1790407787248620 us; error 31 us; skew 500 ppm
Sep 26 07:29:47 dist-test-slave-wdjc krb5kdc[23268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407787, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:29:47.251761 23281 init.cc:377] Logged in from keytab as kudu/127.22.53.254@KRBTEST.COM (short username kudu)
I20260926 07:29:47.252907 23281 webserver.cc:460] Webserver started at http://127.22.53.254:36421/ using document root <none> and password file <none>
I20260926 07:29:47.253568 23281 fs_manager.cc:359] Metadata directory not provided
I20260926 07:29:47.253685 23281 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:29:47.253954 23281 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:29:47.256397 23281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407783529717-22743-0/minicluster-data/master-0/data/instance:
uuid: "ae7e474eeb6546e5a182204db2990bf3"
format_stamp: "Formatted at 2026-09-26 07:29:47 on dist-test-slave-wdjc"
server_key: "6de15a8e3270e5848de4ad2358984f59"
server_key_iv: "5bf06776845aa815bb801b1410385fb4"
server_key_version: "encryptionkey@0"
I20260926 07:29:47.257174 23281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407783529717-22743-0/minicluster-data/master-0/wal/instance:
uuid: "ae7e474eeb6546e5a182204db2990bf3"
format_stamp: "Formatted at 2026-09-26 07:29:47 on dist-test-slave-wdjc"
server_key: "6de15a8e3270e5848de4ad2358984f59"
server_key_iv: "5bf06776845aa815bb801b1410385fb4"
server_key_version: "encryptionkey@0"
I20260926 07:29:47.261581 23281 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 07:29:47.264338 23296 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:47.265923 23281 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:29:47.266139 23281 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407783529717-22743-0/minicluster-data/master-0/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407783529717-22743-0/minicluster-data/master-0/wal
uuid: "ae7e474eeb6546e5a182204db2990bf3"
format_stamp: "Formatted at 2026-09-26 07:29:47 on dist-test-slave-wdjc"
server_key: "6de15a8e3270e5848de4ad2358984f59"
server_key_iv: "5bf06776845aa815bb801b1410385fb4"
server_key_version: "encryptionkey@0"
I20260926 07:29:47.266290 23281 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407783529717-22743-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407783529717-22743-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407783529717-22743-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:29:47.278247 23281 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:29:47.282088 23281 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:29:47.282338 23281 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:29:47.290132 23281 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.254:42721
I20260926 07:29:47.290146 23348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.254:42721 every 8 connection(s)
I20260926 07:29:47.291251 23281 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
I20260926 07:29:47.294432 23349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:29:47.297340 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 23281
I20260926 07:29:47.297470 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407783529717-22743-0/minicluster-data/master-0/wal/instance
I20260926 07:29:47.297740 22743 external_mini_cluster.cc:1428] Setting key 47cb70a4185acfaea7ce870972b26573
I20260926 07:29:47.301518 23349 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae7e474eeb6546e5a182204db2990bf3: Bootstrap starting.
I20260926 07:29:47.304032 23349 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae7e474eeb6546e5a182204db2990bf3: Neither blocks nor log segments found. Creating new log.
I20260926 07:29:47.305104 23349 log.cc:824] T 00000000000000000000000000000000 P ae7e474eeb6546e5a182204db2990bf3: Log is configured to *not* fsync() on all Append() calls
Sep 26 07:29:47 dist-test-slave-wdjc krb5kdc[23268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407787, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.53.254@KRBTEST.COM
I20260926 07:29:47.307694 23349 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae7e474eeb6546e5a182204db2990bf3: No bootstrap required, opened a new log
I20260926 07:29:47.311574 23349 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae7e474eeb6546e5a182204db2990bf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae7e474eeb6546e5a182204db2990bf3" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 42721 } }
I20260926 07:29:47.311828 23349 raft_consensus.cc:374] T 00000000000000000000000000000000 P ae7e474eeb6546e5a182204db2990bf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:29:47.311899 23349 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae7e474eeb6546e5a182204db2990bf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae7e474eeb6546e5a182204db2990bf3, State: Initialized, Role: FOLLOWER
I20260926 07:29:47.312463 23349 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae7e474eeb6546e5a182204db2990bf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae7e474eeb6546e5a182204db2990bf3" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 42721 } }
I20260926 07:29:47.312683 23349 raft_consensus.cc:388] T 00000000000000000000000000000000 P ae7e474eeb6546e5a182204db2990bf3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:29:47.312767 23349 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae7e474eeb6546e5a182204db2990bf3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:29:47.312879 23349 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae7e474eeb6546e5a182204db2990bf3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:29:47.313778 23352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:47.299322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50664 (local address 127.22.53.254:42721)
0926 07:29:47.299876 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:47.299892 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:47.299932 (+ 40us) server_negotiation.cc:407] Connection header received
0926 07:29:47.300693 (+ 761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:47.300714 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:47.301094 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:47.301475 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:47.302338 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:47.303364 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:47.304306 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:47.305071 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:47.305273 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:47.308152 (+ 2879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:47.308158 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:47.308175 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:29:47.308204 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:47.310482 (+ 2278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:47.311042 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:47.311048 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:47.311052 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:47.311174 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:47.311492 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:47.311494 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:47.311495 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:47.312038 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:47.312227 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:47.312508 (+ 281us) server_negotiation.cc:299] Negotiation successful
0926 07:29:47.312922 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":171,"threads_started":1}
I20260926 07:29:47.313941 23349 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae7e474eeb6546e5a182204db2990bf3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae7e474eeb6546e5a182204db2990bf3" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 42721 } }
I20260926 07:29:47.314322 23349 leader_election.cc:302] T 00000000000000000000000000000000 P ae7e474eeb6546e5a182204db2990bf3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae7e474eeb6546e5a182204db2990bf3; no voters:
I20260926 07:29:47.314603 23349 leader_election.cc:288] T 00000000000000000000000000000000 P ae7e474eeb6546e5a182204db2990bf3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:29:47.314738 23354 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae7e474eeb6546e5a182204db2990bf3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:29:47.314953 23354 raft_consensus.cc:686] T 00000000000000000000000000000000 P ae7e474eeb6546e5a182204db2990bf3 [term 1 LEADER]: Becoming Leader. State: Replica: ae7e474eeb6546e5a182204db2990bf3, State: Running, Role: LEADER
I20260926 07:29:47.315264 23354 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae7e474eeb6546e5a182204db2990bf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae7e474eeb6546e5a182204db2990bf3" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 42721 } }
I20260926 07:29:47.315874 23349 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae7e474eeb6546e5a182204db2990bf3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:29:47.317081 23355 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae7e474eeb6546e5a182204db2990bf3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ae7e474eeb6546e5a182204db2990bf3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae7e474eeb6546e5a182204db2990bf3" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 42721 } } }
I20260926 07:29:47.317214 23355 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae7e474eeb6546e5a182204db2990bf3 [sys.catalog]: This master's current role is: LEADER
I20260926 07:29:47.317483 23356 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae7e474eeb6546e5a182204db2990bf3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae7e474eeb6546e5a182204db2990bf3. Latest consensus state: current_term: 1 leader_uuid: "ae7e474eeb6546e5a182204db2990bf3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae7e474eeb6546e5a182204db2990bf3" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 42721 } } }
I20260926 07:29:47.317588 23356 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae7e474eeb6546e5a182204db2990bf3 [sys.catalog]: This master's current role is: LEADER
I20260926 07:29:47.317710 23363 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:29:47.321064 23363 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:29:47.327503 23363 catalog_manager.cc:1347] Generated new cluster ID: a35f5c9d9bac44508a3e68035a94a4be
I20260926 07:29:47.327611 23363 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:29:47.344986 23363 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:29:47.346412 23363 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:29:47.360430 23363 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ae7e474eeb6546e5a182204db2990bf3: Generated new TSK 0
I20260926 07:29:47.361485 23363 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.53.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.53.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.53.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.193.keytab.
Entry for principal kudu/127.22.53.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.193.keytab.
I20260926 07:29:47.399583 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407783529717-22743-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407783529717-22743-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407783529717-22743-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407783529717-22743-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.53.193:0
--local_ip_for_outbound_sockets=127.22.53.193
--webserver_interface=127.22.53.193
--webserver_port=0
--tserver_master_addrs=127.22.53.254:42721
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.193.keytab
--principal=kudu/127.22.53.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:45713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:29:47.525380 23375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:29:47.525767 23375 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:29:47.525856 23375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:29:47.530364 23375 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:29:47.530488 23375 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:29:47.530591 23375 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.53.193
I20260926 07:29:47.536732 23375 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:45713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407783529717-22743-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407783529717-22743-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.193.keytab
--principal=kudu/127.22.53.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.53.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407783529717-22743-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.53.193
--webserver_port=0
--tserver_master_addrs=127.22.53.254:42721
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23375
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.53.193
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407783529717-22743-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:29:47.537930 23375 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:29:47.539113 23375 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 07:29:47.549711 23375 server_base.cc:1034] running on GCE node
W20260926 07:29:47.549626 23380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:29:47.549626 23381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:29:47.549630 23383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:29:47.550336 23375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:29:47.551117 23375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:29:47.552333 23375 hybrid_clock.cc:648] HybridClock initialized: now 1790407787552292 us; error 56 us; skew 500 ppm
Sep 26 07:29:47 dist-test-slave-wdjc krb5kdc[23268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407787, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:29:47.555547 23375 init.cc:377] Logged in from keytab as kudu/127.22.53.193@KRBTEST.COM (short username kudu)
I20260926 07:29:47.556735 23375 webserver.cc:460] Webserver started at http://127.22.53.193:39579/ using document root <none> and password file <none>
I20260926 07:29:47.557339 23375 fs_manager.cc:359] Metadata directory not provided
I20260926 07:29:47.557406 23375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:29:47.557588 23375 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:29:47.559850 23375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407783529717-22743-0/minicluster-data/ts-0/data/instance:
uuid: "05e194e461c044ed878b7422557854a4"
format_stamp: "Formatted at 2026-09-26 07:29:47 on dist-test-slave-wdjc"
server_key: "16f79f49a7e1b1c552c2c104d832ecd1"
server_key_iv: "4656382de156dfa6ca1dfd1de5358969"
server_key_version: "encryptionkey@0"
I20260926 07:29:47.560447 23375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407783529717-22743-0/minicluster-data/ts-0/wal/instance:
uuid: "05e194e461c044ed878b7422557854a4"
format_stamp: "Formatted at 2026-09-26 07:29:47 on dist-test-slave-wdjc"
server_key: "16f79f49a7e1b1c552c2c104d832ecd1"
server_key_iv: "4656382de156dfa6ca1dfd1de5358969"
server_key_version: "encryptionkey@0"
I20260926 07:29:47.564891 23375 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 07:29:47.567927 23390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:47.569458 23375 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:29:47.569629 23375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407783529717-22743-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407783529717-22743-0/minicluster-data/ts-0/wal
uuid: "05e194e461c044ed878b7422557854a4"
format_stamp: "Formatted at 2026-09-26 07:29:47 on dist-test-slave-wdjc"
server_key: "16f79f49a7e1b1c552c2c104d832ecd1"
server_key_iv: "4656382de156dfa6ca1dfd1de5358969"
server_key_version: "encryptionkey@0"
I20260926 07:29:47.569823 23375 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407783529717-22743-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407783529717-22743-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407783529717-22743-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:29:47.582705 23375 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:29:47.586500 23375 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:29:47.586753 23375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:29:47.587422 23375 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:29:47.588359 23375 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:29:47.588434 23375 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:47.588510 23375 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:29:47.588579 23375 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260926 07:29:47.598557 23375 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.193:40599
I20260926 07:29:47.598577 23503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.193:40599 every 8 connection(s)
I20260926 07:29:47.599604 23375 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407783529717-22743-0/minicluster-data/ts-0/data/info.pb
I20260926 07:29:47.608027 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 23375
I20260926 07:29:47.608197 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407783529717-22743-0/minicluster-data/ts-0/wal/instance
I20260926 07:29:47.608501 22743 external_mini_cluster.cc:1428] Setting key 3cddb5638dcb9bef78e8eb2ef218c6fb
Sep 26 07:29:47 dist-test-slave-wdjc krb5kdc[23268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407787, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.193@KRBTEST.COM for kudu/127.22.53.254@KRBTEST.COM
I20260926 07:29:47.616384 23352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:47.601718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.193:44501 (local address 127.22.53.254:42721)
0926 07:29:47.601876 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:47.601881 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:47.602972 (+ 1091us) server_negotiation.cc:407] Connection header received
0926 07:29:47.603907 (+ 935us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:47.603910 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:47.603961 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:47.604048 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:47.605696 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:47.606366 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:47.607740 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:47.608038 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:47.608163 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:47.611522 (+ 3359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:47.611529 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:47.611535 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:29:47.611575 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:47.613321 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:47.614155 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:47.614158 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:47.614159 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:47.614256 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:47.614754 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:47.614757 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:47.614758 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:47.615078 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:47.615170 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:47.616099 (+ 929us) server_negotiation.cc:299] Negotiation successful
0926 07:29:47.616245 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 07:29:47.618033 23506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:47.602113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:42721 (local address 127.22.53.193:44501)
0926 07:29:47.602691 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:47.602767 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0926 07:29:47.603690 (+ 923us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:29:47.604273 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:29:47.604281 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:29:47.604808 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:29:47.605389 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:47.605406 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:47.606601 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:47.606604 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:47.607580 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:47.607591 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:47.608354 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:47.608359 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:47.608644 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:47.609467 (+ 823us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:29:47.609565 (+ 98us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:29:47.611292 (+ 1727us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:29:47.613539 (+ 2247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:29:47.613550 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:29:47.613575 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:29:47.614020 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:29:47.614367 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:29:47.614374 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:29:47.614378 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:29:47.614640 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:29:47.615219 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:29:47.615229 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:29:47.615674 (+ 445us) client_negotiation.cc:769] Sending connection context
0926 07:29:47.616035 (+ 361us) client_negotiation.cc:240] Negotiation successful
0926 07:29:47.616595 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":135,"threads_started":1}
I20260926 07:29:47.619745 23504 heartbeater.cc:347] Connected to a master server at 127.22.53.254:42721
WARNING: no policy specified for kudu/127.22.53.194@KRBTEST.COM; defaulting to no policy
I20260926 07:29:47.620074 23504 heartbeater.cc:464] Registering TS with master...
I20260926 07:29:47.620792 23504 heartbeater.cc:511] Master 127.22.53.254:42721 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.53.194@KRBTEST.COM" created.
I20260926 07:29:47.622946 23313 ts_manager.cc:194] Registered new tserver with Master: 05e194e461c044ed878b7422557854a4 (127.22.53.193:40599)
I20260926 07:29:47.624771 23313 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.53.193@KRBTEST.COM'} at 127.22.53.193:44501
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.53.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.194.keytab.
Entry for principal kudu/127.22.53.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.194.keytab.
I20260926 07:29:47.640038 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407783529717-22743-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407783529717-22743-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407783529717-22743-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407783529717-22743-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.53.194:0
--local_ip_for_outbound_sockets=127.22.53.194
--webserver_interface=127.22.53.194
--webserver_port=0
--tserver_master_addrs=127.22.53.254:42721
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.194.keytab
--principal=kudu/127.22.53.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:45713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:29:47.763060 23509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:29:47.763417 23509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:29:47.763469 23509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:29:47.767942 23509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:29:47.768033 23509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:29:47.768101 23509 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.53.194
I20260926 07:29:47.774627 23509 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:45713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407783529717-22743-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407783529717-22743-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.194.keytab
--principal=kudu/127.22.53.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.53.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407783529717-22743-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.53.194
--webserver_port=0
--tserver_master_addrs=127.22.53.254:42721
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23509
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.53.194
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407783529717-22743-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:29:47.775808 23509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:29:47.777021 23509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:29:47.787905 23517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:29:47.787905 23514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:29:47.787943 23515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:29:47.788331 23509 server_base.cc:1034] running on GCE node
I20260926 07:29:47.788848 23509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:29:47.789623 23509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:29:47.790815 23509 hybrid_clock.cc:648] HybridClock initialized: now 1790407787790776 us; error 56 us; skew 500 ppm
Sep 26 07:29:47 dist-test-slave-wdjc krb5kdc[23268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407787, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:29:47.794052 23509 init.cc:377] Logged in from keytab as kudu/127.22.53.194@KRBTEST.COM (short username kudu)
I20260926 07:29:47.795014 23509 webserver.cc:460] Webserver started at http://127.22.53.194:36601/ using document root <none> and password file <none>
I20260926 07:29:47.795615 23509 fs_manager.cc:359] Metadata directory not provided
I20260926 07:29:47.795693 23509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:29:47.795882 23509 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:29:47.798363 23509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407783529717-22743-0/minicluster-data/ts-1/data/instance:
uuid: "dbdb64d5349f44a3b03cff4f483b2558"
format_stamp: "Formatted at 2026-09-26 07:29:47 on dist-test-slave-wdjc"
server_key: "13520c2a8f7c7699b1f95b3f02f68fa2"
server_key_iv: "28b77d479ba17fd550e85265374ef653"
server_key_version: "encryptionkey@0"
I20260926 07:29:47.799045 23509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407783529717-22743-0/minicluster-data/ts-1/wal/instance:
uuid: "dbdb64d5349f44a3b03cff4f483b2558"
format_stamp: "Formatted at 2026-09-26 07:29:47 on dist-test-slave-wdjc"
server_key: "13520c2a8f7c7699b1f95b3f02f68fa2"
server_key_iv: "28b77d479ba17fd550e85265374ef653"
server_key_version: "encryptionkey@0"
I20260926 07:29:47.803419 23509 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260926 07:29:47.807092 23524 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:47.808743 23509 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 07:29:47.808904 23509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407783529717-22743-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407783529717-22743-0/minicluster-data/ts-1/wal
uuid: "dbdb64d5349f44a3b03cff4f483b2558"
format_stamp: "Formatted at 2026-09-26 07:29:47 on dist-test-slave-wdjc"
server_key: "13520c2a8f7c7699b1f95b3f02f68fa2"
server_key_iv: "28b77d479ba17fd550e85265374ef653"
server_key_version: "encryptionkey@0"
I20260926 07:29:47.809053 23509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407783529717-22743-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407783529717-22743-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407783529717-22743-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:29:47.831805 23509 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:29:47.835888 23509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:29:47.836155 23509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:29:47.836890 23509 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:29:47.837785 23509 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:29:47.837833 23509 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:47.837874 23509 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:29:47.837890 23509 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:47.847718 23509 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.194:43247
I20260926 07:29:47.847934 23637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.194:43247 every 8 connection(s)
I20260926 07:29:47.848892 23509 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407783529717-22743-0/minicluster-data/ts-1/data/info.pb
I20260926 07:29:47.858304 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 23509
I20260926 07:29:47.858412 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407783529717-22743-0/minicluster-data/ts-1/wal/instance
I20260926 07:29:47.858726 22743 external_mini_cluster.cc:1428] Setting key 39782600a5565cb39bd3711528dca588
Sep 26 07:29:47 dist-test-slave-wdjc krb5kdc[23268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407787, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.194@KRBTEST.COM for kudu/127.22.53.254@KRBTEST.COM
I20260926 07:29:47.864992 23352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:47.851015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.194:58375 (local address 127.22.53.254:42721)
0926 07:29:47.851164 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:47.851168 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:47.852113 (+ 945us) server_negotiation.cc:407] Connection header received
0926 07:29:47.853571 (+ 1458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:47.853575 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:47.853666 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:47.853776 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:47.855094 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:47.855641 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:47.856995 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:47.857227 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:47.857351 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:47.860415 (+ 3064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:47.860420 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:47.860423 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:29:47.860452 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:47.862221 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:47.862945 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:47.862952 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:47.862955 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:47.863066 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:47.863428 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:47.863434 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:47.863439 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:47.863730 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:47.863846 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:47.864590 (+ 744us) server_negotiation.cc:299] Negotiation successful
0926 07:29:47.864767 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 07:29:47.866029 23640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:47.851428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:42721 (local address 127.22.53.194:58375)
0926 07:29:47.851936 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:47.851992 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 07:29:47.853329 (+ 1337us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:29:47.853981 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:29:47.853989 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:29:47.854386 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:29:47.854920 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:47.854934 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:47.855793 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:47.855797 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:47.856842 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:47.856854 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:47.857404 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:47.857408 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:47.857594 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:47.858334 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:29:47.858352 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:29:47.860234 (+ 1882us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:29:47.862433 (+ 2199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:29:47.862440 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:29:47.862461 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:29:47.862819 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:29:47.863173 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:29:47.863175 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:29:47.863176 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:29:47.863325 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:29:47.863917 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:29:47.863923 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:29:47.864253 (+ 330us) client_negotiation.cc:769] Sending connection context
0926 07:29:47.864461 (+ 208us) client_negotiation.cc:240] Negotiation successful
0926 07:29:47.864958 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":125,"threads_started":1}
I20260926 07:29:47.867579 23638 heartbeater.cc:347] Connected to a master server at 127.22.53.254:42721
I20260926 07:29:47.867934 23638 heartbeater.cc:464] Registering TS with master...
I20260926 07:29:47.868712 23638 heartbeater.cc:511] Master 127.22.53.254:42721 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.22.53.195@KRBTEST.COM; defaulting to no policy
I20260926 07:29:47.870142 23313 ts_manager.cc:194] Registered new tserver with Master: dbdb64d5349f44a3b03cff4f483b2558 (127.22.53.194:43247)
I20260926 07:29:47.870958 23313 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.53.194@KRBTEST.COM'} at 127.22.53.194:58375
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.53.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.53.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.195.keytab.
Entry for principal kudu/127.22.53.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.195.keytab.
I20260926 07:29:47.887750 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407783529717-22743-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407783529717-22743-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407783529717-22743-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407783529717-22743-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.53.195:0
--local_ip_for_outbound_sockets=127.22.53.195
--webserver_interface=127.22.53.195
--webserver_port=0
--tserver_master_addrs=127.22.53.254:42721
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.195.keytab
--principal=kudu/127.22.53.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:45713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:29:48.007270 23643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:29:48.007591 23643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:29:48.007643 23643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:29:48.012127 23643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:29:48.012233 23643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:29:48.012300 23643 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.53.195
I20260926 07:29:48.018628 23643 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:45713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407783529717-22743-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407783529717-22743-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.195.keytab
--principal=kudu/127.22.53.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.53.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407783529717-22743-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.53.195
--webserver_port=0
--tserver_master_addrs=127.22.53.254:42721
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23643
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.53.195
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407783529717-22743-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:29:48.019865 23643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:29:48.020974 23643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:29:48.031721 23651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:29:48.031667 23649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:29:48.031667 23648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:29:48.031893 23643 server_base.cc:1034] running on GCE node
I20260926 07:29:48.032577 23643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:29:48.033388 23643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:29:48.034596 23643 hybrid_clock.cc:648] HybridClock initialized: now 1790407788034565 us; error 54 us; skew 500 ppm
Sep 26 07:29:48 dist-test-slave-wdjc krb5kdc[23268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407788, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:29:48.037690 23643 init.cc:377] Logged in from keytab as kudu/127.22.53.195@KRBTEST.COM (short username kudu)
I20260926 07:29:48.038662 23643 webserver.cc:460] Webserver started at http://127.22.53.195:36891/ using document root <none> and password file <none>
I20260926 07:29:48.039302 23643 fs_manager.cc:359] Metadata directory not provided
I20260926 07:29:48.039399 23643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:29:48.039644 23643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:29:48.042006 23643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407783529717-22743-0/minicluster-data/ts-2/data/instance:
uuid: "78a44e22908a4a4899630f6a405b8878"
format_stamp: "Formatted at 2026-09-26 07:29:48 on dist-test-slave-wdjc"
server_key: "36adecc337a567ba7ca7309be69a5a70"
server_key_iv: "078063dadd48ee90e7ef980c54f27df4"
server_key_version: "encryptionkey@0"
I20260926 07:29:48.042646 23643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407783529717-22743-0/minicluster-data/ts-2/wal/instance:
uuid: "78a44e22908a4a4899630f6a405b8878"
format_stamp: "Formatted at 2026-09-26 07:29:48 on dist-test-slave-wdjc"
server_key: "36adecc337a567ba7ca7309be69a5a70"
server_key_iv: "078063dadd48ee90e7ef980c54f27df4"
server_key_version: "encryptionkey@0"
I20260926 07:29:48.046962 23643 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 07:29:48.049803 23658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:48.051174 23643 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 07:29:48.051347 23643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407783529717-22743-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407783529717-22743-0/minicluster-data/ts-2/wal
uuid: "78a44e22908a4a4899630f6a405b8878"
format_stamp: "Formatted at 2026-09-26 07:29:48 on dist-test-slave-wdjc"
server_key: "36adecc337a567ba7ca7309be69a5a70"
server_key_iv: "078063dadd48ee90e7ef980c54f27df4"
server_key_version: "encryptionkey@0"
I20260926 07:29:48.051481 23643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407783529717-22743-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407783529717-22743-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407783529717-22743-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:29:48.069525 23643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:29:48.073680 23643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:29:48.073905 23643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:29:48.074615 23643 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:29:48.075632 23643 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:29:48.075711 23643 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:48.075788 23643 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:29:48.075841 23643 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:48.085278 23643 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.195:33809
I20260926 07:29:48.085306 23771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.195:33809 every 8 connection(s)
I20260926 07:29:48.086279 23643 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407783529717-22743-0/minicluster-data/ts-2/data/info.pb
I20260926 07:29:48.095688 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 23643
I20260926 07:29:48.095808 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407783529717-22743-0/minicluster-data/ts-2/wal/instance
I20260926 07:29:48.096120 22743 external_mini_cluster.cc:1428] Setting key 1c87c6e91d8f4d90568d1ab1ccb0705a
Sep 26 07:29:48 dist-test-slave-wdjc krb5kdc[23268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407788, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.195@KRBTEST.COM for kudu/127.22.53.254@KRBTEST.COM
I20260926 07:29:48.101310 23352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:48.088320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.195:44131 (local address 127.22.53.254:42721)
0926 07:29:48.088480 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:48.088484 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:48.089544 (+ 1060us) server_negotiation.cc:407] Connection header received
0926 07:29:48.090542 (+ 998us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:48.090546 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:48.090610 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:48.090728 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:48.092033 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:48.092664 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:48.093900 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:48.094130 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:48.094254 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:48.097200 (+ 2946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:48.097204 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:48.097207 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:29:48.097235 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:48.098837 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:48.099442 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:48.099446 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:48.099448 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:48.099541 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:48.099975 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:48.099978 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:48.099980 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:48.100202 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:48.100315 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:48.100973 (+ 658us) server_negotiation.cc:299] Negotiation successful
0926 07:29:48.101121 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 07:29:48.102226 23774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:48.088803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:42721 (local address 127.22.53.195:44131)
0926 07:29:48.089358 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:48.089411 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 07:29:48.090316 (+ 905us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:29:48.090924 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:29:48.090932 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:29:48.091334 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:29:48.091869 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:48.091883 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:48.092803 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:48.092806 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:48.093751 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:48.093761 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:48.094305 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:48.094308 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:48.094488 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:48.095173 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:29:48.095194 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:29:48.096990 (+ 1796us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:29:48.098985 (+ 1995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:29:48.098992 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:29:48.099012 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:29:48.099329 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:29:48.099644 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:29:48.099648 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:29:48.099650 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:29:48.099881 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:29:48.100331 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:29:48.100340 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:29:48.100713 (+ 373us) client_negotiation.cc:769] Sending connection context
0926 07:29:48.100928 (+ 215us) client_negotiation.cc:240] Negotiation successful
0926 07:29:48.101313 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":147,"threads_started":1}
I20260926 07:29:48.103515 23772 heartbeater.cc:347] Connected to a master server at 127.22.53.254:42721
I20260926 07:29:48.103818 23772 heartbeater.cc:464] Registering TS with master...
I20260926 07:29:48.104398 23772 heartbeater.cc:511] Master 127.22.53.254:42721 requested a full tablet report, sending...
I20260926 07:29:48.105571 23313 ts_manager.cc:194] Registered new tserver with Master: 78a44e22908a4a4899630f6a405b8878 (127.22.53.195:33809)
I20260926 07:29:48.106199 23313 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.53.195@KRBTEST.COM'} at 127.22.53.195:44131
I20260926 07:29:48.110432 22743 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:29:48.122318 23352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:48.114246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50670 (local address 127.22.53.254:42721)
0926 07:29:48.114449 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:48.114456 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:48.114707 (+ 251us) server_negotiation.cc:407] Connection header received
0926 07:29:48.114874 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:48.114878 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:48.114930 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:48.115036 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:48.115856 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:48.116498 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:48.117450 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:48.117664 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:48.117766 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:48.119284 (+ 1518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:48.119287 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:48.119289 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:29:48.119316 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:48.120689 (+ 1373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:48.121133 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:48.121137 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:48.121138 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:48.121233 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:48.121492 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:48.121495 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:48.121497 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:48.121744 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:48.121864 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:48.122079 (+ 215us) server_negotiation.cc:299] Negotiation successful
0926 07:29:48.122189 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260926 07:29:48.127561 23313 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50670:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:29:48.131119 23313 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:29:48.146832 23785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:48.140566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60396 (local address 127.22.53.194:43247)
0926 07:29:48.141294 (+ 728us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:48.141312 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:48.141502 (+ 190us) server_negotiation.cc:407] Connection header received
0926 07:29:48.141587 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:48.141593 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:48.141834 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:48.142052 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:29:48.142437 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:48.143372 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:48.145178 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:48.146049 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:48.146174 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:48.146364 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:48.146542 (+ 178us) server_negotiation.cc:299] Negotiation successful
0926 07:29:48.146662 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":621,"thread_start_us":106,"threads_started":1}
I20260926 07:29:48.146826 23786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:48.140868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.194:43247 (local address 127.0.0.1:60396)
0926 07:29:48.141302 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:48.141315 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:29:48.141399 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:29:48.142234 (+ 835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:29:48.142236 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:29:48.142250 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:29:48.142343 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:48.142349 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:48.143551 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:48.143554 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:48.145049 (+ 1495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:48.145059 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:48.146200 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:48.146204 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:48.146321 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:48.146362 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 07:29:48.146487 (+ 125us) client_negotiation.cc:240] Negotiation successful
0926 07:29:48.146631 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"mutex_wait_us":72,"thread_start_us":90,"threads_started":1}
I20260926 07:29:48.147193 23788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:48.141713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43730 (local address 127.22.53.193:40599)
0926 07:29:48.142405 (+ 692us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:48.142411 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:48.142434 (+ 23us) server_negotiation.cc:407] Connection header received
0926 07:29:48.142510 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:48.142515 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:48.142726 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:48.142876 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:29:48.143130 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:48.144117 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:48.145591 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:48.146202 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:48.146312 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:48.146466 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:48.146899 (+ 433us) server_negotiation.cc:299] Negotiation successful
0926 07:29:48.147033 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":79,"threads_started":1}
I20260926 07:29:48.147084 23783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:48.140419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.195:33809 (local address 127.0.0.1:58106)
0926 07:29:48.140854 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:48.140898 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 07:29:48.141048 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:29:48.142044 (+ 996us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:29:48.142051 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:29:48.142075 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:29:48.142253 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:48.142261 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:48.143679 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:48.143682 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:48.145274 (+ 1592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:48.145286 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:48.146635 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:48.146639 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:48.146770 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:48.146795 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 07:29:48.146861 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 07:29:48.146945 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"mutex_wait_us":175,"thread_start_us":108,"threads_started":1}
I20260926 07:29:48.147584 23784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:48.140396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58106 (local address 127.22.53.195:33809)
0926 07:29:48.141513 (+ 1117us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:48.141526 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:48.141550 (+ 24us) server_negotiation.cc:407] Connection header received
0926 07:29:48.141630 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:48.141637 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:48.141865 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:48.142039 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:29:48.142385 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:48.143529 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:48.145411 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:48.146494 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:48.147143 (+ 649us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:48.147255 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:48.147343 (+ 88us) server_negotiation.cc:299] Negotiation successful
0926 07:29:48.147441 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1012,"thread_start_us":195,"threads_started":1}
I20260926 07:29:48.147838 23787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:48.141294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.193:40599 (local address 127.0.0.1:43730)
0926 07:29:48.142072 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:48.142088 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:29:48.142169 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:29:48.142885 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:29:48.142889 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:29:48.142907 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:29:48.142995 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:48.143000 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:48.144272 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:48.144277 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:48.145478 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:48.145485 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:48.146529 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:48.146533 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:48.146609 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:48.146629 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 07:29:48.146695 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 07:29:48.146784 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":718,"mutex_wait_us":28,"thread_start_us":100,"threads_started":1}
I20260926 07:29:48.150525 23572 tablet_service.cc:1467] Processing CreateTablet for tablet 1d9d252da81b4373a3dc14ac946735d4 (DEFAULT_TABLE table=test-table [id=7a362b107ec549f4bca16ba27de52d0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:29:48.150593 23706 tablet_service.cc:1467] Processing CreateTablet for tablet 1d9d252da81b4373a3dc14ac946735d4 (DEFAULT_TABLE table=test-table [id=7a362b107ec549f4bca16ba27de52d0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:29:48.150593 23438 tablet_service.cc:1467] Processing CreateTablet for tablet 1d9d252da81b4373a3dc14ac946735d4 (DEFAULT_TABLE table=test-table [id=7a362b107ec549f4bca16ba27de52d0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:29:48.151664 23572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d9d252da81b4373a3dc14ac946735d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:29:48.151680 23706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d9d252da81b4373a3dc14ac946735d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:29:48.151675 23438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d9d252da81b4373a3dc14ac946735d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:29:48.159075 23789 tablet_bootstrap.cc:492] T 1d9d252da81b4373a3dc14ac946735d4 P dbdb64d5349f44a3b03cff4f483b2558: Bootstrap starting.
I20260926 07:29:48.160444 23790 tablet_bootstrap.cc:492] T 1d9d252da81b4373a3dc14ac946735d4 P 05e194e461c044ed878b7422557854a4: Bootstrap starting.
I20260926 07:29:48.162010 23789 tablet_bootstrap.cc:654] T 1d9d252da81b4373a3dc14ac946735d4 P dbdb64d5349f44a3b03cff4f483b2558: Neither blocks nor log segments found. Creating new log.
I20260926 07:29:48.162741 23790 tablet_bootstrap.cc:654] T 1d9d252da81b4373a3dc14ac946735d4 P 05e194e461c044ed878b7422557854a4: Neither blocks nor log segments found. Creating new log.
I20260926 07:29:48.163034 23789 log.cc:824] T 1d9d252da81b4373a3dc14ac946735d4 P dbdb64d5349f44a3b03cff4f483b2558: Log is configured to *not* fsync() on all Append() calls
I20260926 07:29:48.163621 23790 log.cc:824] T 1d9d252da81b4373a3dc14ac946735d4 P 05e194e461c044ed878b7422557854a4: Log is configured to *not* fsync() on all Append() calls
I20260926 07:29:48.164039 23791 tablet_bootstrap.cc:492] T 1d9d252da81b4373a3dc14ac946735d4 P 78a44e22908a4a4899630f6a405b8878: Bootstrap starting.
I20260926 07:29:48.165733 23790 tablet_bootstrap.cc:492] T 1d9d252da81b4373a3dc14ac946735d4 P 05e194e461c044ed878b7422557854a4: No bootstrap required, opened a new log
I20260926 07:29:48.165967 23790 ts_tablet_manager.cc:1397] T 1d9d252da81b4373a3dc14ac946735d4 P 05e194e461c044ed878b7422557854a4: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 07:29:48.166603 23791 tablet_bootstrap.cc:654] T 1d9d252da81b4373a3dc14ac946735d4 P 78a44e22908a4a4899630f6a405b8878: Neither blocks nor log segments found. Creating new log.
I20260926 07:29:48.167465 23791 log.cc:824] T 1d9d252da81b4373a3dc14ac946735d4 P 78a44e22908a4a4899630f6a405b8878: Log is configured to *not* fsync() on all Append() calls
I20260926 07:29:48.169234 23789 tablet_bootstrap.cc:492] T 1d9d252da81b4373a3dc14ac946735d4 P dbdb64d5349f44a3b03cff4f483b2558: No bootstrap required, opened a new log
I20260926 07:29:48.169481 23789 ts_tablet_manager.cc:1397] T 1d9d252da81b4373a3dc14ac946735d4 P dbdb64d5349f44a3b03cff4f483b2558: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
I20260926 07:29:48.169714 23791 tablet_bootstrap.cc:492] T 1d9d252da81b4373a3dc14ac946735d4 P 78a44e22908a4a4899630f6a405b8878: No bootstrap required, opened a new log
I20260926 07:29:48.169922 23791 ts_tablet_manager.cc:1397] T 1d9d252da81b4373a3dc14ac946735d4 P 78a44e22908a4a4899630f6a405b8878: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 07:29:48.170457 23790 raft_consensus.cc:348] T 1d9d252da81b4373a3dc14ac946735d4 P 05e194e461c044ed878b7422557854a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78a44e22908a4a4899630f6a405b8878" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 33809 } } peers { permanent_uuid: "dbdb64d5349f44a3b03cff4f483b2558" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 43247 } } peers { permanent_uuid: "05e194e461c044ed878b7422557854a4" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 40599 } }
I20260926 07:29:48.170789 23790 raft_consensus.cc:374] T 1d9d252da81b4373a3dc14ac946735d4 P 05e194e461c044ed878b7422557854a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:29:48.170848 23790 raft_consensus.cc:729] T 1d9d252da81b4373a3dc14ac946735d4 P 05e194e461c044ed878b7422557854a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05e194e461c044ed878b7422557854a4, State: Initialized, Role: FOLLOWER
I20260926 07:29:48.171380 23790 consensus_queue.cc:260] T 1d9d252da81b4373a3dc14ac946735d4 P 05e194e461c044ed878b7422557854a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78a44e22908a4a4899630f6a405b8878" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 33809 } } peers { permanent_uuid: "dbdb64d5349f44a3b03cff4f483b2558" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 43247 } } peers { permanent_uuid: "05e194e461c044ed878b7422557854a4" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 40599 } }
I20260926 07:29:48.172492 23504 heartbeater.cc:503] Master 127.22.53.254:42721 was elected leader, sending a full tablet report...
I20260926 07:29:48.172503 23790 ts_tablet_manager.cc:1428] T 1d9d252da81b4373a3dc14ac946735d4 P 05e194e461c044ed878b7422557854a4: Time spent starting tablet: real 0.006s user 0.004s sys 0.003s
I20260926 07:29:48.173884 23789 raft_consensus.cc:348] T 1d9d252da81b4373a3dc14ac946735d4 P dbdb64d5349f44a3b03cff4f483b2558 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78a44e22908a4a4899630f6a405b8878" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 33809 } } peers { permanent_uuid: "dbdb64d5349f44a3b03cff4f483b2558" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 43247 } } peers { permanent_uuid: "05e194e461c044ed878b7422557854a4" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 40599 } }
I20260926 07:29:48.174211 23789 raft_consensus.cc:374] T 1d9d252da81b4373a3dc14ac946735d4 P dbdb64d5349f44a3b03cff4f483b2558 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:29:48.174271 23789 raft_consensus.cc:729] T 1d9d252da81b4373a3dc14ac946735d4 P dbdb64d5349f44a3b03cff4f483b2558 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbdb64d5349f44a3b03cff4f483b2558, State: Initialized, Role: FOLLOWER
I20260926 07:29:48.174768 23789 consensus_queue.cc:260] T 1d9d252da81b4373a3dc14ac946735d4 P dbdb64d5349f44a3b03cff4f483b2558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78a44e22908a4a4899630f6a405b8878" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 33809 } } peers { permanent_uuid: "dbdb64d5349f44a3b03cff4f483b2558" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 43247 } } peers { permanent_uuid: "05e194e461c044ed878b7422557854a4" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 40599 } }
I20260926 07:29:48.175585 23638 heartbeater.cc:503] Master 127.22.53.254:42721 was elected leader, sending a full tablet report...
I20260926 07:29:48.175372 23791 raft_consensus.cc:348] T 1d9d252da81b4373a3dc14ac946735d4 P 78a44e22908a4a4899630f6a405b8878 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78a44e22908a4a4899630f6a405b8878" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 33809 } } peers { permanent_uuid: "dbdb64d5349f44a3b03cff4f483b2558" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 43247 } } peers { permanent_uuid: "05e194e461c044ed878b7422557854a4" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 40599 } }
I20260926 07:29:48.175684 23791 raft_consensus.cc:374] T 1d9d252da81b4373a3dc14ac946735d4 P 78a44e22908a4a4899630f6a405b8878 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:29:48.175733 23791 raft_consensus.cc:729] T 1d9d252da81b4373a3dc14ac946735d4 P 78a44e22908a4a4899630f6a405b8878 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78a44e22908a4a4899630f6a405b8878, State: Initialized, Role: FOLLOWER
I20260926 07:29:48.175997 23789 ts_tablet_manager.cc:1428] T 1d9d252da81b4373a3dc14ac946735d4 P dbdb64d5349f44a3b03cff4f483b2558: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260926 07:29:48.176239 23791 consensus_queue.cc:260] T 1d9d252da81b4373a3dc14ac946735d4 P 78a44e22908a4a4899630f6a405b8878 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78a44e22908a4a4899630f6a405b8878" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 33809 } } peers { permanent_uuid: "dbdb64d5349f44a3b03cff4f483b2558" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 43247 } } peers { permanent_uuid: "05e194e461c044ed878b7422557854a4" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 40599 } }
I20260926 07:29:48.176982 23772 heartbeater.cc:503] Master 127.22.53.254:42721 was elected leader, sending a full tablet report...
I20260926 07:29:48.178419 23791 ts_tablet_manager.cc:1428] T 1d9d252da81b4373a3dc14ac946735d4 P 78a44e22908a4a4899630f6a405b8878: Time spent starting tablet: real 0.008s user 0.006s sys 0.001s
I20260926 07:29:48.195751 23797 raft_consensus.cc:482] T 1d9d252da81b4373a3dc14ac946735d4 P 78a44e22908a4a4899630f6a405b8878 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:29:48.196017 23797 raft_consensus.cc:504] T 1d9d252da81b4373a3dc14ac946735d4 P 78a44e22908a4a4899630f6a405b8878 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78a44e22908a4a4899630f6a405b8878" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 33809 } } peers { permanent_uuid: "dbdb64d5349f44a3b03cff4f483b2558" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 43247 } } peers { permanent_uuid: "05e194e461c044ed878b7422557854a4" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 40599 } }
I20260926 07:29:48.197309 23797 leader_election.cc:288] T 1d9d252da81b4373a3dc14ac946735d4 P 78a44e22908a4a4899630f6a405b8878 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dbdb64d5349f44a3b03cff4f483b2558 (127.22.53.194:43247), 05e194e461c044ed878b7422557854a4 (127.22.53.193:40599)
I20260926 07:29:48.201623 23788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:48.197781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.195:50941 (local address 127.22.53.193:40599)
0926 07:29:48.197908 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:48.197911 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:48.197926 (+ 15us) server_negotiation.cc:407] Connection header received
0926 07:29:48.197977 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:48.197979 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:48.198027 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:48.198107 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:29:48.198451 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:48.199164 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:48.200380 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:48.200979 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:48.201085 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:48.201129 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:48.201404 (+ 275us) server_negotiation.cc:299] Negotiation successful
0926 07:29:48.201488 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 07:29:48.201612 23774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:48.197587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.193:40599 (local address 127.22.53.195:50941)
0926 07:29:48.197726 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:48.197744 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 07:29:48.197875 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:29:48.198160 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:29:48.198166 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:29:48.198191 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:29:48.198320 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:48.198335 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:48.199275 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:48.199279 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:48.200251 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:48.200261 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:48.201103 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:48.201107 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:48.201239 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:48.201267 (+ 28us) client_negotiation.cc:769] Sending connection context
0926 07:29:48.201349 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 07:29:48.201443 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260926 07:29:48.201927 23798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:48.197587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.194:43247 (local address 127.22.53.195:50147)
0926 07:29:48.198345 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:48.198356 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 07:29:48.198475 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:29:48.198732 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:29:48.198735 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:29:48.198753 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:29:48.198869 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:48.198875 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:48.199735 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:48.199737 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:48.200686 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:48.200697 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:48.201459 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:48.201463 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:48.201570 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:48.201592 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 07:29:48.201674 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 07:29:48.201749 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":696,"mutex_wait_us":369,"thread_start_us":172,"threads_started":1}
I20260926 07:29:48.201931 23785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:48.197767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.195:50147 (local address 127.22.53.194:43247)
0926 07:29:48.197932 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:48.197936 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:48.198442 (+ 506us) server_negotiation.cc:407] Connection header received
0926 07:29:48.198587 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:48.198589 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:48.198631 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:48.198703 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:29:48.198976 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:48.199605 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:48.200808 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:48.201334 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:48.201445 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:48.201488 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:48.201716 (+ 228us) server_negotiation.cc:299] Negotiation successful
0926 07:29:48.201798 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 07:29:48.202448 23458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d9d252da81b4373a3dc14ac946735d4" candidate_uuid: "78a44e22908a4a4899630f6a405b8878" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05e194e461c044ed878b7422557854a4" is_pre_election: true
I20260926 07:29:48.202867 23458 raft_consensus.cc:2454] T 1d9d252da81b4373a3dc14ac946735d4 P 05e194e461c044ed878b7422557854a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78a44e22908a4a4899630f6a405b8878 in term 0.
I20260926 07:29:48.203115 23592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d9d252da81b4373a3dc14ac946735d4" candidate_uuid: "78a44e22908a4a4899630f6a405b8878" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbdb64d5349f44a3b03cff4f483b2558" is_pre_election: true
I20260926 07:29:48.203452 23592 raft_consensus.cc:2454] T 1d9d252da81b4373a3dc14ac946735d4 P dbdb64d5349f44a3b03cff4f483b2558 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78a44e22908a4a4899630f6a405b8878 in term 0.
I20260926 07:29:48.203516 23661 leader_election.cc:302] T 1d9d252da81b4373a3dc14ac946735d4 P 78a44e22908a4a4899630f6a405b8878 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05e194e461c044ed878b7422557854a4, 78a44e22908a4a4899630f6a405b8878; no voters:
I20260926 07:29:48.203867 23797 raft_consensus.cc:2781] T 1d9d252da81b4373a3dc14ac946735d4 P 78a44e22908a4a4899630f6a405b8878 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:29:48.203955 23797 raft_consensus.cc:482] T 1d9d252da81b4373a3dc14ac946735d4 P 78a44e22908a4a4899630f6a405b8878 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:29:48.203998 23797 raft_consensus.cc:3037] T 1d9d252da81b4373a3dc14ac946735d4 P 78a44e22908a4a4899630f6a405b8878 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:29:48.205116 23797 raft_consensus.cc:504] T 1d9d252da81b4373a3dc14ac946735d4 P 78a44e22908a4a4899630f6a405b8878 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78a44e22908a4a4899630f6a405b8878" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 33809 } } peers { permanent_uuid: "dbdb64d5349f44a3b03cff4f483b2558" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 43247 } } peers { permanent_uuid: "05e194e461c044ed878b7422557854a4" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 40599 } }
I20260926 07:29:48.205556 23797 leader_election.cc:288] T 1d9d252da81b4373a3dc14ac946735d4 P 78a44e22908a4a4899630f6a405b8878 [CANDIDATE]: Term 1 election: Requested vote from peers dbdb64d5349f44a3b03cff4f483b2558 (127.22.53.194:43247), 05e194e461c044ed878b7422557854a4 (127.22.53.193:40599)
I20260926 07:29:48.205966 23592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d9d252da81b4373a3dc14ac946735d4" candidate_uuid: "78a44e22908a4a4899630f6a405b8878" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbdb64d5349f44a3b03cff4f483b2558"
I20260926 07:29:48.206089 23458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d9d252da81b4373a3dc14ac946735d4" candidate_uuid: "78a44e22908a4a4899630f6a405b8878" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05e194e461c044ed878b7422557854a4"
I20260926 07:29:48.206138 23592 raft_consensus.cc:3037] T 1d9d252da81b4373a3dc14ac946735d4 P dbdb64d5349f44a3b03cff4f483b2558 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:29:48.206284 23458 raft_consensus.cc:3037] T 1d9d252da81b4373a3dc14ac946735d4 P 05e194e461c044ed878b7422557854a4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:29:48.207293 23592 raft_consensus.cc:2454] T 1d9d252da81b4373a3dc14ac946735d4 P dbdb64d5349f44a3b03cff4f483b2558 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78a44e22908a4a4899630f6a405b8878 in term 1.
I20260926 07:29:48.207450 23458 raft_consensus.cc:2454] T 1d9d252da81b4373a3dc14ac946735d4 P 05e194e461c044ed878b7422557854a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78a44e22908a4a4899630f6a405b8878 in term 1.
I20260926 07:29:48.207692 23660 leader_election.cc:302] T 1d9d252da81b4373a3dc14ac946735d4 P 78a44e22908a4a4899630f6a405b8878 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78a44e22908a4a4899630f6a405b8878, dbdb64d5349f44a3b03cff4f483b2558; no voters:
I20260926 07:29:48.207856 23797 raft_consensus.cc:2781] T 1d9d252da81b4373a3dc14ac946735d4 P 78a44e22908a4a4899630f6a405b8878 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:29:48.208132 23797 raft_consensus.cc:686] T 1d9d252da81b4373a3dc14ac946735d4 P 78a44e22908a4a4899630f6a405b8878 [term 1 LEADER]: Becoming Leader. State: Replica: 78a44e22908a4a4899630f6a405b8878, State: Running, Role: LEADER
I20260926 07:29:48.208498 23797 consensus_queue.cc:237] T 1d9d252da81b4373a3dc14ac946735d4 P 78a44e22908a4a4899630f6a405b8878 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78a44e22908a4a4899630f6a405b8878" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 33809 } } peers { permanent_uuid: "dbdb64d5349f44a3b03cff4f483b2558" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 43247 } } peers { permanent_uuid: "05e194e461c044ed878b7422557854a4" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 40599 } }
I20260926 07:29:48.211895 23312 catalog_manager.cc:5495] T 1d9d252da81b4373a3dc14ac946735d4 P 78a44e22908a4a4899630f6a405b8878 reported cstate change: term changed from 0 to 1, leader changed from <none> to 78a44e22908a4a4899630f6a405b8878 (127.22.53.195). New cstate: current_term: 1 leader_uuid: "78a44e22908a4a4899630f6a405b8878" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78a44e22908a4a4899630f6a405b8878" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 33809 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dbdb64d5349f44a3b03cff4f483b2558" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 43247 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05e194e461c044ed878b7422557854a4" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 40599 } health_report { overall_health: UNKNOWN } } }
Sep 26 07:29:48 dist-test-slave-wdjc krb5kdc[23268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407787, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.53.193@KRBTEST.COM
I20260926 07:29:48.234742 23788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:48.221442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43746 (local address 127.22.53.193:40599)
0926 07:29:48.221606 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:48.221611 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:48.221741 (+ 130us) server_negotiation.cc:407] Connection header received
0926 07:29:48.221912 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:48.221916 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:48.221967 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:48.222054 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:48.222844 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:48.223886 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:48.225473 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:48.225942 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:48.226106 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:48.230205 (+ 4099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:48.230227 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:48.230249 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:29:48.230320 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:48.232426 (+ 2106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:48.233124 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:48.233133 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:48.233134 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:48.233246 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:48.233594 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:48.233598 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:48.233600 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:48.233913 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:48.234187 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:48.234380 (+ 193us) server_negotiation.cc:299] Negotiation successful
0926 07:29:48.234541 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
Sep 26 07:29:48 dist-test-slave-wdjc krb5kdc[23268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407788, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 07:29:48 dist-test-slave-wdjc krb5kdc[23268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407788, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.22.53.193@KRBTEST.COM
I20260926 07:29:48.263465 23788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:48.254433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43752 (local address 127.22.53.193:40599)
0926 07:29:48.254602 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:48.254609 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:48.254878 (+ 269us) server_negotiation.cc:407] Connection header received
0926 07:29:48.254998 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:48.255001 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:48.255050 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:48.255127 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:48.255950 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:48.256663 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:48.257489 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:48.257719 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:48.257797 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:48.260299 (+ 2502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:48.260302 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:48.260304 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:29:48.260336 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:48.261765 (+ 1429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:48.262201 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:48.262203 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:48.262205 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:48.262284 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:48.262518 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:48.262520 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:48.262521 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:48.262692 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:48.262799 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:48.263139 (+ 340us) server_negotiation.cc:299] Negotiation successful
0926 07:29:48.263258 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
Sep 26 07:29:48 dist-test-slave-wdjc krb5kdc[23268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407788, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 07:29:48 dist-test-slave-wdjc krb5kdc[23268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407788, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.53.193@KRBTEST.COM
I20260926 07:29:48.290813 23788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:48.281618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43754 (local address 127.22.53.193:40599)
0926 07:29:48.281798 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:48.281803 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:48.281978 (+ 175us) server_negotiation.cc:407] Connection header received
0926 07:29:48.282171 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:48.282174 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:48.282230 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:48.282322 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:48.283075 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:48.283590 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:48.284477 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:48.284784 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:48.284903 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:48.287571 (+ 2668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:48.287574 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:48.287576 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:29:48.287605 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:48.289052 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:48.289496 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:48.289501 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:48.289502 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:48.289591 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:48.289900 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:48.289903 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:48.289905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:48.290089 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:48.290237 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:48.290505 (+ 268us) server_negotiation.cc:299] Negotiation successful
0926 07:29:48.290637 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260926 07:29:48.292201 23418 tablet_service.cc:2908] Rejecting scan request for tablet 1d9d252da81b4373a3dc14ac946735d4: Uninitialized: clean time has not yet been initialized
I20260926 07:29:48.294443 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 23375
I20260926 07:29:48.302081 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 23509
I20260926 07:29:48.307838 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 23643
I20260926 07:29:48.314008 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 23281
2026-09-26T07:29:48Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2430 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407783529717-22743-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:29:48 dist-test-slave-wdjc krb5kdc[23813](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:29:48 dist-test-slave-wdjc krb5kdc[23813](info): set up 2 sockets
Sep 26 07:29:48 dist-test-slave-wdjc krb5kdc[23813](info): commencing operation
krb5kdc: starting...
W20260926 07:29:49.372970 22743 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407783529717-22743-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407783529717-22743-0/krb5kdc/test-user.keytab.
Sep 26 07:29:49 dist-test-slave-wdjc krb5kdc[23813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407789, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:29:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:29:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.53.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.53.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.53.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab.
Entry for principal kudu/127.22.53.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab.
I20260926 07:29:49.530256 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407783529717-22743-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407783529717-22743-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407783529717-22743-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.53.254:44201
--webserver_interface=127.22.53.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab
--principal=kudu/127.22.53.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:34705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.53.254:44201
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:29:49.648818 23826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:29:49.649227 23826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:29:49.649309 23826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:29:49.653731 23826 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:29:49.653860 23826 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 07:29:49.653892 23826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:29:49.653937 23826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:29:49.653975 23826 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:29:49.654009 23826 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:29:49.659942 23826 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:34705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407783529717-22743-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407783529717-22743-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.53.254:44201
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab
--principal=kudu/127.22.53.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.53.254:44201
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.53.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23826
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407783529717-22743-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:29:49.661262 23826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:29:49.662410 23826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:29:49.671964 23832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:29:49.672027 23826 server_base.cc:1034] running on GCE node
W20260926 07:29:49.671964 23834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:29:49.671969 23831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:29:49.673014 23826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:29:49.674451 23826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:29:49.675696 23826 hybrid_clock.cc:648] HybridClock initialized: now 1790407789675669 us; error 51 us; skew 500 ppm
Sep 26 07:29:49 dist-test-slave-wdjc krb5kdc[23813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407789, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:29:49.682780 23826 init.cc:377] Logged in from keytab as kudu/127.22.53.254@KRBTEST.COM (short username kudu)
I20260926 07:29:49.683988 23826 webserver.cc:460] Webserver started at http://127.22.53.254:39537/ using document root <none> and password file <none>
I20260926 07:29:49.684773 23826 fs_manager.cc:359] Metadata directory not provided
I20260926 07:29:49.684883 23826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:29:49.685148 23826 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:29:49.687471 23826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407783529717-22743-0/minicluster-data/master-0/data/instance:
uuid: "7a0dca45b0b949f39066155a4f9f160f"
format_stamp: "Formatted at 2026-09-26 07:29:49 on dist-test-slave-wdjc"
server_key: "ab7236efd6354e06289af35043f338e2"
server_key_iv: "b9c34cd946cd1d5631e61dcbc497454b"
server_key_version: "encryptionkey@0"
I20260926 07:29:49.688158 23826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407783529717-22743-0/minicluster-data/master-0/wal/instance:
uuid: "7a0dca45b0b949f39066155a4f9f160f"
format_stamp: "Formatted at 2026-09-26 07:29:49 on dist-test-slave-wdjc"
server_key: "ab7236efd6354e06289af35043f338e2"
server_key_iv: "b9c34cd946cd1d5631e61dcbc497454b"
server_key_version: "encryptionkey@0"
I20260926 07:29:49.692683 23826 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.002s
I20260926 07:29:49.695704 23841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:49.697286 23826 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:29:49.697502 23826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407783529717-22743-0/minicluster-data/master-0/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407783529717-22743-0/minicluster-data/master-0/wal
uuid: "7a0dca45b0b949f39066155a4f9f160f"
format_stamp: "Formatted at 2026-09-26 07:29:49 on dist-test-slave-wdjc"
server_key: "ab7236efd6354e06289af35043f338e2"
server_key_iv: "b9c34cd946cd1d5631e61dcbc497454b"
server_key_version: "encryptionkey@0"
I20260926 07:29:49.697669 23826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407783529717-22743-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407783529717-22743-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407783529717-22743-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:29:49.708336 23826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:29:49.712603 23826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:29:49.712873 23826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:29:49.721937 23826 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.254:44201
I20260926 07:29:49.721961 23903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.254:44201 every 8 connection(s)
I20260926 07:29:49.723062 23826 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
I20260926 07:29:49.726468 23904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:29:49.727715 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 23826
I20260926 07:29:49.727836 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407783529717-22743-0/minicluster-data/master-0/wal/instance
I20260926 07:29:49.728133 22743 external_mini_cluster.cc:1428] Setting key 81581cc5fc1f642c02b0d97a69d912c8
I20260926 07:29:49.733901 23904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a0dca45b0b949f39066155a4f9f160f: Bootstrap starting.
I20260926 07:29:49.736971 23904 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a0dca45b0b949f39066155a4f9f160f: Neither blocks nor log segments found. Creating new log.
I20260926 07:29:49.737911 23904 log.cc:824] T 00000000000000000000000000000000 P 7a0dca45b0b949f39066155a4f9f160f: Log is configured to *not* fsync() on all Append() calls
Sep 26 07:29:49 dist-test-slave-wdjc krb5kdc[23813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407789, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.53.254@KRBTEST.COM
I20260926 07:29:49.740686 23904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a0dca45b0b949f39066155a4f9f160f: No bootstrap required, opened a new log
I20260926 07:29:49.745327 23907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:49.729733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58954 (local address 127.22.53.254:44201)
0926 07:29:49.730208 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:49.730227 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:49.730266 (+ 39us) server_negotiation.cc:407] Connection header received
0926 07:29:49.730956 (+ 690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:49.730977 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:49.731337 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:49.731674 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:49.732718 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:49.733749 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:49.734949 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:49.735473 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:49.735630 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:49.738698 (+ 3068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:49.738704 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:49.738719 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:29:49.738749 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:49.741587 (+ 2838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:49.742139 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:49.742145 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:49.742152 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:49.742317 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:49.742720 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:49.742723 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:49.742725 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:49.743312 (+ 587us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:49.743519 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:49.743824 (+ 305us) server_negotiation.cc:299] Negotiation successful
0926 07:29:49.744173 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":161,"threads_started":1}
I20260926 07:29:49.745188 23904 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a0dca45b0b949f39066155a4f9f160f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a0dca45b0b949f39066155a4f9f160f" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 44201 } }
I20260926 07:29:49.745483 23904 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7a0dca45b0b949f39066155a4f9f160f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:29:49.745534 23904 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a0dca45b0b949f39066155a4f9f160f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a0dca45b0b949f39066155a4f9f160f, State: Initialized, Role: FOLLOWER
I20260926 07:29:49.746001 23904 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a0dca45b0b949f39066155a4f9f160f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a0dca45b0b949f39066155a4f9f160f" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 44201 } }
I20260926 07:29:49.746178 23904 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7a0dca45b0b949f39066155a4f9f160f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:29:49.746253 23904 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a0dca45b0b949f39066155a4f9f160f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:29:49.746363 23904 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a0dca45b0b949f39066155a4f9f160f [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:29:49.747435 23904 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a0dca45b0b949f39066155a4f9f160f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a0dca45b0b949f39066155a4f9f160f" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 44201 } }
I20260926 07:29:49.747835 23904 leader_election.cc:302] T 00000000000000000000000000000000 P 7a0dca45b0b949f39066155a4f9f160f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7a0dca45b0b949f39066155a4f9f160f; no voters:
I20260926 07:29:49.748242 23904 leader_election.cc:288] T 00000000000000000000000000000000 P 7a0dca45b0b949f39066155a4f9f160f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:29:49.748374 23909 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7a0dca45b0b949f39066155a4f9f160f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:29:49.748649 23909 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7a0dca45b0b949f39066155a4f9f160f [term 1 LEADER]: Becoming Leader. State: Replica: 7a0dca45b0b949f39066155a4f9f160f, State: Running, Role: LEADER
I20260926 07:29:49.748984 23909 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a0dca45b0b949f39066155a4f9f160f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a0dca45b0b949f39066155a4f9f160f" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 44201 } }
I20260926 07:29:49.749632 23904 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a0dca45b0b949f39066155a4f9f160f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:29:49.750707 23911 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a0dca45b0b949f39066155a4f9f160f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a0dca45b0b949f39066155a4f9f160f. Latest consensus state: current_term: 1 leader_uuid: "7a0dca45b0b949f39066155a4f9f160f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a0dca45b0b949f39066155a4f9f160f" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 44201 } } }
I20260926 07:29:49.750850 23911 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a0dca45b0b949f39066155a4f9f160f [sys.catalog]: This master's current role is: LEADER
I20260926 07:29:49.751161 23910 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a0dca45b0b949f39066155a4f9f160f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7a0dca45b0b949f39066155a4f9f160f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a0dca45b0b949f39066155a4f9f160f" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 44201 } } }
I20260926 07:29:49.751260 23910 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a0dca45b0b949f39066155a4f9f160f [sys.catalog]: This master's current role is: LEADER
I20260926 07:29:49.752024 23918 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:29:49.755092 23918 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:29:49.761901 23918 catalog_manager.cc:1347] Generated new cluster ID: b36893575dd24f61ab6d66ed2ea307d3
I20260926 07:29:49.762004 23918 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:29:49.772354 23918 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:29:49.773464 23918 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:29:49.778396 23918 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7a0dca45b0b949f39066155a4f9f160f: Generated new TSK 0
I20260926 07:29:49.779258 23918 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.53.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.53.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.53.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.193.keytab.
Entry for principal kudu/127.22.53.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.193.keytab.
I20260926 07:29:49.812418 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407783529717-22743-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407783529717-22743-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407783529717-22743-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407783529717-22743-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.53.193:0
--local_ip_for_outbound_sockets=127.22.53.193
--webserver_interface=127.22.53.193
--webserver_port=0
--tserver_master_addrs=127.22.53.254:44201
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.193.keytab
--principal=kudu/127.22.53.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:34705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:29:49.933444 23930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:29:49.933758 23930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:29:49.933807 23930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:29:49.938169 23930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:29:49.938247 23930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:29:49.938341 23930 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 07:29:49.938411 23930 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.53.193
I20260926 07:29:49.944114 23930 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:34705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407783529717-22743-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407783529717-22743-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.193.keytab
--principal=kudu/127.22.53.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.53.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407783529717-22743-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.53.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.53.254:44201
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.53.193
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407783529717-22743-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:29:49.945423 23930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:29:49.946460 23930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:29:49.957053 23935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:29:49.957053 23936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:29:49.957053 23938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:29:49.957468 23930 server_base.cc:1034] running on GCE node
I20260926 07:29:49.957937 23930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:29:49.958683 23930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:29:49.959910 23930 hybrid_clock.cc:648] HybridClock initialized: now 1790407789959898 us; error 35 us; skew 500 ppm
Sep 26 07:29:49 dist-test-slave-wdjc krb5kdc[23813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407789, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:29:49.963025 23930 init.cc:377] Logged in from keytab as kudu/127.22.53.193@KRBTEST.COM (short username kudu)
I20260926 07:29:49.964038 23930 webserver.cc:460] Webserver started at http://127.22.53.193:46197/ using document root <none> and password file <none>
I20260926 07:29:49.964778 23930 fs_manager.cc:359] Metadata directory not provided
I20260926 07:29:49.964885 23930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:29:49.965137 23930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:29:49.967427 23930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407783529717-22743-0/minicluster-data/ts-0/data/instance:
uuid: "066cd56b67e94fc4932f012941b09759"
format_stamp: "Formatted at 2026-09-26 07:29:49 on dist-test-slave-wdjc"
server_key: "6528da79e673c15777d23996837dba8d"
server_key_iv: "e2e871ecf9d6b097971958596151c51a"
server_key_version: "encryptionkey@0"
I20260926 07:29:49.968091 23930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407783529717-22743-0/minicluster-data/ts-0/wal/instance:
uuid: "066cd56b67e94fc4932f012941b09759"
format_stamp: "Formatted at 2026-09-26 07:29:49 on dist-test-slave-wdjc"
server_key: "6528da79e673c15777d23996837dba8d"
server_key_iv: "e2e871ecf9d6b097971958596151c51a"
server_key_version: "encryptionkey@0"
I20260926 07:29:49.972460 23930 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260926 07:29:49.975453 23945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:49.977137 23930 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:29:49.977347 23930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407783529717-22743-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407783529717-22743-0/minicluster-data/ts-0/wal
uuid: "066cd56b67e94fc4932f012941b09759"
format_stamp: "Formatted at 2026-09-26 07:29:49 on dist-test-slave-wdjc"
server_key: "6528da79e673c15777d23996837dba8d"
server_key_iv: "e2e871ecf9d6b097971958596151c51a"
server_key_version: "encryptionkey@0"
I20260926 07:29:49.977505 23930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407783529717-22743-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407783529717-22743-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407783529717-22743-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:29:49.989955 23930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:29:49.993477 23930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:29:49.993718 23930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:29:49.995286 23930 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:29:49.995371 23930 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:49.995440 23930 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:29:49.995486 23930 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:50.007061 23930 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.193:37391
I20260926 07:29:50.007114 24060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.193:37391 every 8 connection(s)
Sep 26 07:29:50 dist-test-slave-wdjc krb5kdc[23813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407789, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.193@KRBTEST.COM for kudu/127.22.53.254@KRBTEST.COM
I20260926 07:29:50.008296 23930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407783529717-22743-0/minicluster-data/ts-0/data/info.pb
I20260926 07:29:50.009836 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 23930
I20260926 07:29:50.009939 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407783529717-22743-0/minicluster-data/ts-0/wal/instance
I20260926 07:29:50.010221 22743 external_mini_cluster.cc:1428] Setting key 4f02f053cc59eb7d5df813bca95790a7
I20260926 07:29:50.012822 23907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:49.996349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.193:43327 (local address 127.22.53.254:44201)
0926 07:29:49.996480 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:49.996483 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:49.997747 (+ 1264us) server_negotiation.cc:407] Connection header received
0926 07:29:49.998949 (+ 1202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:49.998955 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:49.999023 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:49.999088 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:50.001069 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:50.001849 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:50.003785 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:50.004030 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:50.004122 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:50.008154 (+ 4032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:50.008158 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:50.008160 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:29:50.008193 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:50.010341 (+ 2148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:50.011049 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:50.011053 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:50.011055 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:50.011156 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:50.011560 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:50.011563 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:50.011565 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:50.011769 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:50.011881 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:50.012447 (+ 566us) server_negotiation.cc:299] Negotiation successful
0926 07:29:50.012642 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 07:29:50.014055 23954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:49.996992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:44201 (local address 127.22.53.193:43327)
0926 07:29:49.997539 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:49.997584 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 07:29:49.998673 (+ 1089us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:29:49.999564 (+ 891us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:29:49.999572 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:29:50.000105 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:29:50.000898 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:50.000936 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:50.001994 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:50.002001 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:50.003644 (+ 1643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:50.003656 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:50.004221 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:50.004228 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:50.004475 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:50.005355 (+ 880us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:29:50.005384 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:29:50.007881 (+ 2497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:29:50.010497 (+ 2616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:29:50.010505 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:29:50.010524 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:29:50.010913 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:29:50.011267 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:29:50.011271 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:29:50.011273 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:29:50.011443 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:29:50.011883 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:29:50.011893 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:29:50.012184 (+ 291us) client_negotiation.cc:769] Sending connection context
0926 07:29:50.012393 (+ 209us) client_negotiation.cc:240] Negotiation successful
0926 07:29:50.012801 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":154,"threads_started":1}
I20260926 07:29:50.015767 24061 heartbeater.cc:347] Connected to a master server at 127.22.53.254:44201
I20260926 07:29:50.015938 24061 heartbeater.cc:464] Registering TS with master...
I20260926 07:29:50.016630 24061 heartbeater.cc:511] Master 127.22.53.254:44201 requested a full tablet report, sending...
I20260926 07:29:50.018715 23858 ts_manager.cc:194] Registered new tserver with Master: 066cd56b67e94fc4932f012941b09759 (127.22.53.193:37391)
I20260926 07:29:50.020608 23858 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.53.193@KRBTEST.COM'} at 127.22.53.193:43327
WARNING: no policy specified for kudu/127.22.53.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.53.194@KRBTEST.COM" created.
I20260926 07:29:50.026695 23907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:50.018271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.193:53887 (local address 127.22.53.254:44201)
0926 07:29:50.018419 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:50.018423 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:50.018648 (+ 225us) server_negotiation.cc:407] Connection header received
0926 07:29:50.018801 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:50.018805 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:50.018859 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:50.018933 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:50.019487 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:50.020355 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:50.021399 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:50.021632 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:50.021749 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:50.023048 (+ 1299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:50.023052 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:50.023055 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:29:50.023087 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:50.024858 (+ 1771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:50.025335 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:50.025342 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:50.025346 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:50.025449 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:50.025725 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:50.025731 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:50.025735 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:50.026003 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:50.026117 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:50.026442 (+ 325us) server_negotiation.cc:299] Negotiation successful
0926 07:29:50.026554 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":285,"server-negotiator.queue_time_us":60}
I20260926 07:29:50.026865 24068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:50.018203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:44201 (local address 127.22.53.193:53887)
0926 07:29:50.018562 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:50.018574 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 07:29:50.018694 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:29:50.019055 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:29:50.019057 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:29:50.019249 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:29:50.019358 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:50.019364 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:50.020499 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:50.020502 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:50.021267 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:50.021278 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:50.021769 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:50.021772 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:50.021876 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:50.022447 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:29:50.022460 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:29:50.022891 (+ 431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:29:50.025008 (+ 2117us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:29:50.025011 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:29:50.025013 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:29:50.025224 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:29:50.025549 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:29:50.025551 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:29:50.025553 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:29:50.025631 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:29:50.026114 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:29:50.026118 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:29:50.026251 (+ 133us) client_negotiation.cc:769] Sending connection context
0926 07:29:50.026382 (+ 131us) client_negotiation.cc:240] Negotiation successful
0926 07:29:50.026571 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":120,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.53.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.194.keytab.
Entry for principal kudu/127.22.53.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.194.keytab.
I20260926 07:29:50.041811 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407783529717-22743-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407783529717-22743-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407783529717-22743-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407783529717-22743-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.53.194:0
--local_ip_for_outbound_sockets=127.22.53.194
--webserver_interface=127.22.53.194
--webserver_port=0
--tserver_master_addrs=127.22.53.254:44201
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.194.keytab
--principal=kudu/127.22.53.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:34705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:29:50.160171 24070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:29:50.160499 24070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:29:50.160570 24070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:29:50.164975 24070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:29:50.165056 24070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:29:50.165099 24070 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 07:29:50.165172 24070 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.53.194
I20260926 07:29:50.170976 24070 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:34705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407783529717-22743-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407783529717-22743-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.194.keytab
--principal=kudu/127.22.53.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.53.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407783529717-22743-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.53.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.53.254:44201
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24070
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.53.194
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407783529717-22743-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:29:50.172230 24070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:29:50.173427 24070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:29:50.184437 24078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:29:50.184430 24076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:29:50.184677 24070 server_base.cc:1034] running on GCE node
W20260926 07:29:50.184432 24075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:29:50.185170 24070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:29:50.185930 24070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:29:50.187115 24070 hybrid_clock.cc:648] HybridClock initialized: now 1790407790187095 us; error 35 us; skew 500 ppm
Sep 26 07:29:50 dist-test-slave-wdjc krb5kdc[23813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407790, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:29:50.190348 24070 init.cc:377] Logged in from keytab as kudu/127.22.53.194@KRBTEST.COM (short username kudu)
I20260926 07:29:50.191445 24070 webserver.cc:460] Webserver started at http://127.22.53.194:38163/ using document root <none> and password file <none>
I20260926 07:29:50.192072 24070 fs_manager.cc:359] Metadata directory not provided
I20260926 07:29:50.192144 24070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:29:50.192691 24070 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:29:50.195600 24070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407783529717-22743-0/minicluster-data/ts-1/data/instance:
uuid: "f704d805fef340d68c74e444508c238d"
format_stamp: "Formatted at 2026-09-26 07:29:50 on dist-test-slave-wdjc"
server_key: "9e6298376bbda87675b99a461fbe07e6"
server_key_iv: "16c710339d3d2d5a944b01dedb83ec16"
server_key_version: "encryptionkey@0"
I20260926 07:29:50.196280 24070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407783529717-22743-0/minicluster-data/ts-1/wal/instance:
uuid: "f704d805fef340d68c74e444508c238d"
format_stamp: "Formatted at 2026-09-26 07:29:50 on dist-test-slave-wdjc"
server_key: "9e6298376bbda87675b99a461fbe07e6"
server_key_iv: "16c710339d3d2d5a944b01dedb83ec16"
server_key_version: "encryptionkey@0"
I20260926 07:29:50.200451 24070 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 07:29:50.203402 24085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:50.204909 24070 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 07:29:50.205045 24070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407783529717-22743-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407783529717-22743-0/minicluster-data/ts-1/wal
uuid: "f704d805fef340d68c74e444508c238d"
format_stamp: "Formatted at 2026-09-26 07:29:50 on dist-test-slave-wdjc"
server_key: "9e6298376bbda87675b99a461fbe07e6"
server_key_iv: "16c710339d3d2d5a944b01dedb83ec16"
server_key_version: "encryptionkey@0"
I20260926 07:29:50.205169 24070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407783529717-22743-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407783529717-22743-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407783529717-22743-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:29:50.237368 24070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:29:50.240983 24070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:29:50.241163 24070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:29:50.242688 24070 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:29:50.242779 24070 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:50.242856 24070 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:29:50.242908 24070 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 07:29:50 dist-test-slave-wdjc krb5kdc[23813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407790, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.194@KRBTEST.COM for kudu/127.22.53.254@KRBTEST.COM
I20260926 07:29:50.254944 24070 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.194:36981
I20260926 07:29:50.254999 24200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.194:36981 every 8 connection(s)
I20260926 07:29:50.255919 24070 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407783529717-22743-0/minicluster-data/ts-1/data/info.pb
I20260926 07:29:50.258956 23907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:50.243711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.194:54279 (local address 127.22.53.254:44201)
0926 07:29:50.243879 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:50.243885 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:50.244796 (+ 911us) server_negotiation.cc:407] Connection header received
0926 07:29:50.245895 (+ 1099us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:50.245898 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:50.245965 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:50.246054 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:50.248003 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:50.248998 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:50.250676 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:50.251148 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:50.251294 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:50.254894 (+ 3600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:50.254898 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:50.254901 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:29:50.254938 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:50.256725 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:50.257291 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:50.257295 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:50.257297 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:50.257398 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:50.257763 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:50.257768 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:50.257770 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:50.257997 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:50.258115 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:50.258651 (+ 536us) server_negotiation.cc:299] Negotiation successful
0926 07:29:50.258788 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 07:29:50.259645 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 24070
I20260926 07:29:50.259743 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407783529717-22743-0/minicluster-data/ts-1/wal/instance
I20260926 07:29:50.260038 22743 external_mini_cluster.cc:1428] Setting key b448b21d4197825c5f93b06c35942dcc
I20260926 07:29:50.260035 24094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:50.244109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:44201 (local address 127.22.53.194:54279)
0926 07:29:50.244622 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:50.244675 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 07:29:50.245655 (+ 980us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:29:50.246448 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:29:50.246460 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:29:50.247065 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:29:50.247780 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:50.247804 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:50.249164 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:50.249167 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:50.250486 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:50.250499 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:50.251531 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:50.251536 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:50.251780 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:50.252595 (+ 815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:29:50.252628 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:29:50.254631 (+ 2003us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:29:50.256855 (+ 2224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:29:50.256861 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:29:50.256876 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:29:50.257170 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:29:50.257502 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:29:50.257505 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:29:50.257507 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:29:50.257662 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:29:50.258117 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:29:50.258126 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:29:50.258396 (+ 270us) client_negotiation.cc:769] Sending connection context
0926 07:29:50.258606 (+ 210us) client_negotiation.cc:240] Negotiation successful
0926 07:29:50.258956 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":157,"threads_started":1}
I20260926 07:29:50.261906 24201 heartbeater.cc:347] Connected to a master server at 127.22.53.254:44201
I20260926 07:29:50.262105 24201 heartbeater.cc:464] Registering TS with master...
I20260926 07:29:50.262847 24201 heartbeater.cc:511] Master 127.22.53.254:44201 requested a full tablet report, sending...
I20260926 07:29:50.264214 23857 ts_manager.cc:194] Registered new tserver with Master: f704d805fef340d68c74e444508c238d (127.22.53.194:36981)
I20260926 07:29:50.265056 23857 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.53.194@KRBTEST.COM'} at 127.22.53.194:54279
WARNING: no policy specified for kudu/127.22.53.195@KRBTEST.COM; defaulting to no policy
I20260926 07:29:50.271978 23907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:50.264369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.194:34491 (local address 127.22.53.254:44201)
0926 07:29:50.264479 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:50.264483 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:50.264499 (+ 16us) server_negotiation.cc:407] Connection header received
0926 07:29:50.264679 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:50.264682 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:50.264727 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:50.264814 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:50.265505 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:50.266094 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:50.266955 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:50.267197 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:50.267315 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:50.268615 (+ 1300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:50.268618 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:50.268621 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:29:50.268647 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:50.270137 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:50.270612 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:50.270616 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:50.270618 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:50.270704 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:50.270999 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:50.271004 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:50.271005 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:50.271254 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:50.271373 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:50.271668 (+ 295us) server_negotiation.cc:299] Negotiation successful
0926 07:29:50.271784 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 07:29:50.271986 24208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:50.264009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:44201 (local address 127.22.53.194:34491)
0926 07:29:50.264386 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:50.264409 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 07:29:50.264526 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:29:50.264857 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:29:50.264861 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:29:50.265163 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:29:50.265334 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:50.265347 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:50.266233 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:50.266236 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:50.266825 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:50.266835 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:50.267324 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:50.267327 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:50.267425 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:50.268015 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:29:50.268028 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:29:50.268433 (+ 405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:29:50.270270 (+ 1837us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:29:50.270275 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:29:50.270278 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:29:50.270503 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:29:50.270804 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:29:50.270808 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:29:50.270810 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:29:50.270904 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:29:50.271361 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:29:50.271364 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:29:50.271480 (+ 116us) client_negotiation.cc:769] Sending connection context
0926 07:29:50.271611 (+ 131us) client_negotiation.cc:240] Negotiation successful
0926 07:29:50.271784 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":118,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.53.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.53.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.195.keytab.
Entry for principal kudu/127.22.53.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.195.keytab.
I20260926 07:29:50.292004 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407783529717-22743-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407783529717-22743-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407783529717-22743-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407783529717-22743-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.53.195:0
--local_ip_for_outbound_sockets=127.22.53.195
--webserver_interface=127.22.53.195
--webserver_port=0
--tserver_master_addrs=127.22.53.254:44201
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.195.keytab
--principal=kudu/127.22.53.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:34705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:29:50.411455 24210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:29:50.411767 24210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:29:50.411815 24210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:29:50.416194 24210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:29:50.416272 24210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:29:50.416316 24210 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 07:29:50.416352 24210 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.53.195
I20260926 07:29:50.422032 24210 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:34705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407783529717-22743-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407783529717-22743-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.195.keytab
--principal=kudu/127.22.53.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.53.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407783529717-22743-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.53.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.53.254:44201
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24210
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.53.195
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407783529717-22743-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:29:50.423168 24210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:29:50.424202 24210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:29:50.434382 24215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:29:50.434438 24216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:29:50.434438 24218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:29:50.434643 24210 server_base.cc:1034] running on GCE node
I20260926 07:29:50.435168 24210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:29:50.435943 24210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:29:50.437142 24210 hybrid_clock.cc:648] HybridClock initialized: now 1790407790437102 us; error 55 us; skew 500 ppm
Sep 26 07:29:50 dist-test-slave-wdjc krb5kdc[23813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407790, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:29:50.440127 24210 init.cc:377] Logged in from keytab as kudu/127.22.53.195@KRBTEST.COM (short username kudu)
I20260926 07:29:50.441088 24210 webserver.cc:460] Webserver started at http://127.22.53.195:36933/ using document root <none> and password file <none>
I20260926 07:29:50.441640 24210 fs_manager.cc:359] Metadata directory not provided
I20260926 07:29:50.441701 24210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:29:50.441866 24210 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:29:50.443982 24210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407783529717-22743-0/minicluster-data/ts-2/data/instance:
uuid: "0083994f1c12460eb29337ad7b217e91"
format_stamp: "Formatted at 2026-09-26 07:29:50 on dist-test-slave-wdjc"
server_key: "8a5ff3ce7973e4aee60b108816de03d6"
server_key_iv: "a7d17eddcfe28641333da5becd1ca38b"
server_key_version: "encryptionkey@0"
I20260926 07:29:50.444628 24210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407783529717-22743-0/minicluster-data/ts-2/wal/instance:
uuid: "0083994f1c12460eb29337ad7b217e91"
format_stamp: "Formatted at 2026-09-26 07:29:50 on dist-test-slave-wdjc"
server_key: "8a5ff3ce7973e4aee60b108816de03d6"
server_key_iv: "a7d17eddcfe28641333da5becd1ca38b"
server_key_version: "encryptionkey@0"
I20260926 07:29:50.449282 24210 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 07:29:50.452294 24225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:50.453790 24210 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:29:50.453946 24210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407783529717-22743-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407783529717-22743-0/minicluster-data/ts-2/wal
uuid: "0083994f1c12460eb29337ad7b217e91"
format_stamp: "Formatted at 2026-09-26 07:29:50 on dist-test-slave-wdjc"
server_key: "8a5ff3ce7973e4aee60b108816de03d6"
server_key_iv: "a7d17eddcfe28641333da5becd1ca38b"
server_key_version: "encryptionkey@0"
I20260926 07:29:50.454085 24210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407783529717-22743-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407783529717-22743-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407783529717-22743-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:29:50.471153 24210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:29:50.475054 24210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:29:50.475276 24210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:29:50.476907 24210 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:29:50.476962 24210 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:50.477005 24210 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:29:50.477057 24210 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 07:29:50 dist-test-slave-wdjc krb5kdc[23813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407790, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.195@KRBTEST.COM for kudu/127.22.53.254@KRBTEST.COM
I20260926 07:29:50.488842 24210 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.195:45383
I20260926 07:29:50.489106 24340 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.195:45383 every 8 connection(s)
I20260926 07:29:50.489997 24210 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407783529717-22743-0/minicluster-data/ts-2/data/info.pb
I20260926 07:29:50.491858 23907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:50.477858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.195:57263 (local address 127.22.53.254:44201)
0926 07:29:50.478043 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:50.478048 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:50.478933 (+ 885us) server_negotiation.cc:407] Connection header received
0926 07:29:50.480020 (+ 1087us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:50.480030 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:50.480119 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:50.480188 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:50.481933 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:50.482587 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:50.484472 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:50.484857 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:50.484989 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:50.487754 (+ 2765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:50.487757 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:50.487760 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:29:50.487797 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:50.489603 (+ 1806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:50.490241 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:50.490245 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:50.490246 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:50.490330 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:50.490764 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:50.490770 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:50.490773 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:50.491017 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:50.491106 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:50.491615 (+ 509us) server_negotiation.cc:299] Negotiation successful
0926 07:29:50.491732 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 07:29:50.492834 24234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:50.478263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:44201 (local address 127.22.53.195:57263)
0926 07:29:50.478740 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:50.478802 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0926 07:29:50.479752 (+ 950us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:29:50.480720 (+ 968us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:29:50.480730 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:29:50.481173 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:29:50.481749 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:50.481764 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:50.482734 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:50.482738 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:50.484346 (+ 1608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:50.484360 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:50.485047 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:50.485051 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:50.485260 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:50.485966 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:29:50.485983 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:29:50.487553 (+ 1570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:29:50.489732 (+ 2179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:29:50.489738 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:29:50.489754 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:29:50.490126 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:29:50.490436 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:29:50.490442 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:29:50.490445 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:29:50.490630 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:29:50.491134 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:29:50.491139 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:29:50.491391 (+ 252us) client_negotiation.cc:769] Sending connection context
0926 07:29:50.491558 (+ 167us) client_negotiation.cc:240] Negotiation successful
0926 07:29:50.491824 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":143,"threads_started":1}
I20260926 07:29:50.494091 24341 heartbeater.cc:347] Connected to a master server at 127.22.53.254:44201
I20260926 07:29:50.494262 24341 heartbeater.cc:464] Registering TS with master...
I20260926 07:29:50.494841 24341 heartbeater.cc:511] Master 127.22.53.254:44201 requested a full tablet report, sending...
I20260926 07:29:50.496052 23858 ts_manager.cc:194] Registered new tserver with Master: 0083994f1c12460eb29337ad7b217e91 (127.22.53.195:45383)
I20260926 07:29:50.496861 23858 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.53.195@KRBTEST.COM'} at 127.22.53.195:57263
I20260926 07:29:50.499689 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 24210
I20260926 07:29:50.499848 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407783529717-22743-0/minicluster-data/ts-2/wal/instance
I20260926 07:29:50.500129 22743 external_mini_cluster.cc:1428] Setting key a075d9e45359ce84cc213aa23cf429fc
I20260926 07:29:50.502592 22743 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:29:50.504943 23907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:50.496978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.195:33901 (local address 127.22.53.254:44201)
0926 07:29:50.497090 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:50.497094 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:50.497353 (+ 259us) server_negotiation.cc:407] Connection header received
0926 07:29:50.497507 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:50.497509 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:50.497559 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:50.497648 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:50.498118 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:50.498635 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:50.499509 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:50.499735 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:50.499841 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:50.501260 (+ 1419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:50.501264 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:50.501267 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:29:50.501297 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:50.503007 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:50.503489 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:50.503495 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:50.503499 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:50.503596 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:50.503878 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:50.503884 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:50.503889 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:50.504161 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:50.504285 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:50.504626 (+ 341us) server_negotiation.cc:299] Negotiation successful
0926 07:29:50.504797 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 07:29:50.505054 24347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:50.496887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:44201 (local address 127.22.53.195:33901)
0926 07:29:50.497261 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:50.497273 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 07:29:50.497407 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:29:50.497666 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:29:50.497670 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:29:50.497896 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:29:50.498001 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:50.498007 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:50.498788 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:50.498790 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:50.499384 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:50.499392 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:50.499860 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:50.499863 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:50.499964 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:50.500581 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:29:50.500603 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:29:50.501054 (+ 451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:29:50.503148 (+ 2094us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:29:50.503151 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:29:50.503152 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:29:50.503377 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:29:50.503698 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:29:50.503701 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:29:50.503702 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:29:50.503785 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:29:50.504300 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:29:50.504303 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:29:50.504400 (+ 97us) client_negotiation.cc:769] Sending connection context
0926 07:29:50.504499 (+ 99us) client_negotiation.cc:240] Negotiation successful
0926 07:29:50.504802 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":119,"threads_started":1}
Sep 26 07:29:50 dist-test-slave-wdjc krb5kdc[23813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407790, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 07:29:50 dist-test-slave-wdjc krb5kdc[23813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407790, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.22.53.254@KRBTEST.COM
I20260926 07:29:50.531666 23907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:50.522171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58960 (local address 127.22.53.254:44201)
0926 07:29:50.522336 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:50.522340 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:50.522355 (+ 15us) server_negotiation.cc:407] Connection header received
0926 07:29:50.522401 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:50.522403 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:50.522455 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:50.522537 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:50.523320 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:50.523855 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:50.524810 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:50.525031 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:50.525128 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:50.527868 (+ 2740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:50.527871 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:50.527873 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:29:50.527904 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:50.529588 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:50.530143 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:50.530146 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:50.530148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:50.530242 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:50.530608 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:50.530611 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:50.530612 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:50.530829 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:50.530951 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:50.531283 (+ 332us) server_negotiation.cc:299] Negotiation successful
0926 07:29:50.531447 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 07:29:50.535211 23858 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58960:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:29:50.538456 23858 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:29:50.554620 24360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:50.548000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.193:37391 (local address 127.0.0.1:47934)
0926 07:29:50.548907 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:50.548920 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:29:50.548996 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:29:50.549749 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:29:50.549759 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:29:50.549785 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:29:50.549936 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:50.549943 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:50.551533 (+ 1590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:50.551537 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:50.553106 (+ 1569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:50.553116 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:50.554054 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:50.554057 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:50.554168 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:50.554207 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 07:29:50.554331 (+ 124us) client_negotiation.cc:240] Negotiation successful
0926 07:29:50.554464 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":823,"mutex_wait_us":348,"thread_start_us":130,"threads_started":1}
I20260926 07:29:50.554620 24362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:50.548796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47934 (local address 127.22.53.193:37391)
0926 07:29:50.549149 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:50.549157 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:50.549180 (+ 23us) server_negotiation.cc:407] Connection header received
0926 07:29:50.549281 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:50.549290 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:50.549578 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:50.549741 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:29:50.550062 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:50.551390 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:50.553241 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:50.553925 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:50.554040 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:50.554219 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:50.554363 (+ 144us) server_negotiation.cc:299] Negotiation successful
0926 07:29:50.554467 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":56,"threads_started":1}
I20260926 07:29:50.555094 24361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:50.548750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.195:45383 (local address 127.0.0.1:57132)
0926 07:29:50.549536 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:50.549549 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:29:50.549641 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:29:50.550338 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:29:50.550341 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:29:50.550359 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:29:50.550463 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:50.550469 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:50.551798 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:50.551801 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:50.553177 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:50.553189 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:50.554471 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:50.554475 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:50.554699 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:50.554797 (+ 98us) client_negotiation.cc:769] Sending connection context
0926 07:29:50.554868 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 07:29:50.554951 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":724,"mutex_wait_us":170,"thread_start_us":102,"threads_started":1}
I20260926 07:29:50.555362 24359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:50.548038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45356 (local address 127.22.53.194:36981)
0926 07:29:50.549258 (+ 1220us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:50.549272 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:50.549293 (+ 21us) server_negotiation.cc:407] Connection header received
0926 07:29:50.549371 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:50.549376 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:50.549652 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:50.549821 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:29:50.550204 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:50.551159 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:50.553328 (+ 2169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:50.554148 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:50.554256 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:50.554408 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:50.555117 (+ 709us) server_negotiation.cc:299] Negotiation successful
0926 07:29:50.555228 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1111,"thread_start_us":94,"threads_started":1}
I20260926 07:29:50.555498 24363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:50.549171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57132 (local address 127.22.53.195:45383)
0926 07:29:50.549910 (+ 739us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:50.549927 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:50.549944 (+ 17us) server_negotiation.cc:407] Connection header received
0926 07:29:50.550034 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:50.550039 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:50.550217 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:50.550365 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:29:50.550549 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:50.551700 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:50.553328 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:50.554148 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:50.554256 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:50.554406 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:50.555260 (+ 854us) server_negotiation.cc:299] Negotiation successful
0926 07:29:50.555352 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":629,"thread_start_us":110,"threads_started":1}
I20260926 07:29:50.555748 24358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:50.547931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.194:36981 (local address 127.0.0.1:45356)
0926 07:29:50.548308 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:50.548359 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 07:29:50.548488 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:29:50.549953 (+ 1465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:29:50.549956 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:29:50.549975 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:29:50.550078 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:50.550083 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:50.551280 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:50.551285 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:50.553177 (+ 1892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:50.553189 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:50.554471 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:50.554475 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:50.554703 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:50.554797 (+ 94us) client_negotiation.cc:769] Sending connection context
0926 07:29:50.554868 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 07:29:50.555633 (+ 765us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"mutex_wait_us":247,"thread_start_us":119,"threads_started":1}
I20260926 07:29:50.558302 23995 tablet_service.cc:1467] Processing CreateTablet for tablet d216a30d60604e44bea6d9a395508aa2 (DEFAULT_TABLE table=test-table [id=aad6677a503e4c5b95908776ed0385b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:29:50.558311 24135 tablet_service.cc:1467] Processing CreateTablet for tablet d216a30d60604e44bea6d9a395508aa2 (DEFAULT_TABLE table=test-table [id=aad6677a503e4c5b95908776ed0385b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:29:50.558964 24275 tablet_service.cc:1467] Processing CreateTablet for tablet d216a30d60604e44bea6d9a395508aa2 (DEFAULT_TABLE table=test-table [id=aad6677a503e4c5b95908776ed0385b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:29:50.559500 23995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d216a30d60604e44bea6d9a395508aa2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:29:50.559500 24135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d216a30d60604e44bea6d9a395508aa2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:29:50.559900 24275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d216a30d60604e44bea6d9a395508aa2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:29:50.567015 24364 tablet_bootstrap.cc:492] T d216a30d60604e44bea6d9a395508aa2 P 0083994f1c12460eb29337ad7b217e91: Bootstrap starting.
I20260926 07:29:50.567909 24365 tablet_bootstrap.cc:492] T d216a30d60604e44bea6d9a395508aa2 P 066cd56b67e94fc4932f012941b09759: Bootstrap starting.
I20260926 07:29:50.568895 24366 tablet_bootstrap.cc:492] T d216a30d60604e44bea6d9a395508aa2 P f704d805fef340d68c74e444508c238d: Bootstrap starting.
I20260926 07:29:50.569631 24364 tablet_bootstrap.cc:654] T d216a30d60604e44bea6d9a395508aa2 P 0083994f1c12460eb29337ad7b217e91: Neither blocks nor log segments found. Creating new log.
I20260926 07:29:50.569864 24365 tablet_bootstrap.cc:654] T d216a30d60604e44bea6d9a395508aa2 P 066cd56b67e94fc4932f012941b09759: Neither blocks nor log segments found. Creating new log.
I20260926 07:29:50.570552 24365 log.cc:824] T d216a30d60604e44bea6d9a395508aa2 P 066cd56b67e94fc4932f012941b09759: Log is configured to *not* fsync() on all Append() calls
I20260926 07:29:50.570556 24364 log.cc:824] T d216a30d60604e44bea6d9a395508aa2 P 0083994f1c12460eb29337ad7b217e91: Log is configured to *not* fsync() on all Append() calls
I20260926 07:29:50.571038 24366 tablet_bootstrap.cc:654] T d216a30d60604e44bea6d9a395508aa2 P f704d805fef340d68c74e444508c238d: Neither blocks nor log segments found. Creating new log.
I20260926 07:29:50.571852 24366 log.cc:824] T d216a30d60604e44bea6d9a395508aa2 P f704d805fef340d68c74e444508c238d: Log is configured to *not* fsync() on all Append() calls
I20260926 07:29:50.572757 24365 tablet_bootstrap.cc:492] T d216a30d60604e44bea6d9a395508aa2 P 066cd56b67e94fc4932f012941b09759: No bootstrap required, opened a new log
I20260926 07:29:50.572988 24365 ts_tablet_manager.cc:1397] T d216a30d60604e44bea6d9a395508aa2 P 066cd56b67e94fc4932f012941b09759: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260926 07:29:50.573912 24364 tablet_bootstrap.cc:492] T d216a30d60604e44bea6d9a395508aa2 P 0083994f1c12460eb29337ad7b217e91: No bootstrap required, opened a new log
I20260926 07:29:50.573912 24366 tablet_bootstrap.cc:492] T d216a30d60604e44bea6d9a395508aa2 P f704d805fef340d68c74e444508c238d: No bootstrap required, opened a new log
I20260926 07:29:50.574115 24366 ts_tablet_manager.cc:1397] T d216a30d60604e44bea6d9a395508aa2 P f704d805fef340d68c74e444508c238d: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 07:29:50.574106 24364 ts_tablet_manager.cc:1397] T d216a30d60604e44bea6d9a395508aa2 P 0083994f1c12460eb29337ad7b217e91: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260926 07:29:50.576663 24365 raft_consensus.cc:348] T d216a30d60604e44bea6d9a395508aa2 P 066cd56b67e94fc4932f012941b09759 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f704d805fef340d68c74e444508c238d" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 36981 } } peers { permanent_uuid: "066cd56b67e94fc4932f012941b09759" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 37391 } } peers { permanent_uuid: "0083994f1c12460eb29337ad7b217e91" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 45383 } }
I20260926 07:29:50.576948 24365 raft_consensus.cc:374] T d216a30d60604e44bea6d9a395508aa2 P 066cd56b67e94fc4932f012941b09759 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:29:50.577010 24365 raft_consensus.cc:729] T d216a30d60604e44bea6d9a395508aa2 P 066cd56b67e94fc4932f012941b09759 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 066cd56b67e94fc4932f012941b09759, State: Initialized, Role: FOLLOWER
I20260926 07:29:50.577502 24365 consensus_queue.cc:260] T d216a30d60604e44bea6d9a395508aa2 P 066cd56b67e94fc4932f012941b09759 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f704d805fef340d68c74e444508c238d" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 36981 } } peers { permanent_uuid: "066cd56b67e94fc4932f012941b09759" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 37391 } } peers { permanent_uuid: "0083994f1c12460eb29337ad7b217e91" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 45383 } }
I20260926 07:29:50.578188 24061 heartbeater.cc:503] Master 127.22.53.254:44201 was elected leader, sending a full tablet report...
I20260926 07:29:50.578125 24366 raft_consensus.cc:348] T d216a30d60604e44bea6d9a395508aa2 P f704d805fef340d68c74e444508c238d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f704d805fef340d68c74e444508c238d" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 36981 } } peers { permanent_uuid: "066cd56b67e94fc4932f012941b09759" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 37391 } } peers { permanent_uuid: "0083994f1c12460eb29337ad7b217e91" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 45383 } }
I20260926 07:29:50.578119 24364 raft_consensus.cc:348] T d216a30d60604e44bea6d9a395508aa2 P 0083994f1c12460eb29337ad7b217e91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f704d805fef340d68c74e444508c238d" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 36981 } } peers { permanent_uuid: "066cd56b67e94fc4932f012941b09759" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 37391 } } peers { permanent_uuid: "0083994f1c12460eb29337ad7b217e91" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 45383 } }
I20260926 07:29:50.578385 24364 raft_consensus.cc:374] T d216a30d60604e44bea6d9a395508aa2 P 0083994f1c12460eb29337ad7b217e91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:29:50.578385 24366 raft_consensus.cc:374] T d216a30d60604e44bea6d9a395508aa2 P f704d805fef340d68c74e444508c238d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:29:50.578449 24366 raft_consensus.cc:729] T d216a30d60604e44bea6d9a395508aa2 P f704d805fef340d68c74e444508c238d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f704d805fef340d68c74e444508c238d, State: Initialized, Role: FOLLOWER
I20260926 07:29:50.578449 24364 raft_consensus.cc:729] T d216a30d60604e44bea6d9a395508aa2 P 0083994f1c12460eb29337ad7b217e91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0083994f1c12460eb29337ad7b217e91, State: Initialized, Role: FOLLOWER
I20260926 07:29:50.578893 24364 consensus_queue.cc:260] T d216a30d60604e44bea6d9a395508aa2 P 0083994f1c12460eb29337ad7b217e91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f704d805fef340d68c74e444508c238d" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 36981 } } peers { permanent_uuid: "066cd56b67e94fc4932f012941b09759" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 37391 } } peers { permanent_uuid: "0083994f1c12460eb29337ad7b217e91" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 45383 } }
I20260926 07:29:50.579789 24365 ts_tablet_manager.cc:1428] T d216a30d60604e44bea6d9a395508aa2 P 066cd56b67e94fc4932f012941b09759: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260926 07:29:50.579924 24364 ts_tablet_manager.cc:1428] T d216a30d60604e44bea6d9a395508aa2 P 0083994f1c12460eb29337ad7b217e91: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260926 07:29:50.580088 24366 consensus_queue.cc:260] T d216a30d60604e44bea6d9a395508aa2 P f704d805fef340d68c74e444508c238d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f704d805fef340d68c74e444508c238d" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 36981 } } peers { permanent_uuid: "066cd56b67e94fc4932f012941b09759" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 37391 } } peers { permanent_uuid: "0083994f1c12460eb29337ad7b217e91" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 45383 } }
I20260926 07:29:50.580250 24341 heartbeater.cc:503] Master 127.22.53.254:44201 was elected leader, sending a full tablet report...
I20260926 07:29:50.580871 24201 heartbeater.cc:503] Master 127.22.53.254:44201 was elected leader, sending a full tablet report...
I20260926 07:29:50.581111 24366 ts_tablet_manager.cc:1428] T d216a30d60604e44bea6d9a395508aa2 P f704d805fef340d68c74e444508c238d: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
W20260926 07:29:50.741132 24342 tablet.cc:2369] T d216a30d60604e44bea6d9a395508aa2 P 0083994f1c12460eb29337ad7b217e91: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 07:29:50.757897 24202 tablet.cc:2369] T d216a30d60604e44bea6d9a395508aa2 P f704d805fef340d68c74e444508c238d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 07:29:50.760008 24062 tablet.cc:2369] T d216a30d60604e44bea6d9a395508aa2 P 066cd56b67e94fc4932f012941b09759: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:29:50.810846 24371 raft_consensus.cc:482] T d216a30d60604e44bea6d9a395508aa2 P 0083994f1c12460eb29337ad7b217e91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:29:50.811116 24371 raft_consensus.cc:504] T d216a30d60604e44bea6d9a395508aa2 P 0083994f1c12460eb29337ad7b217e91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f704d805fef340d68c74e444508c238d" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 36981 } } peers { permanent_uuid: "066cd56b67e94fc4932f012941b09759" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 37391 } } peers { permanent_uuid: "0083994f1c12460eb29337ad7b217e91" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 45383 } }
I20260926 07:29:50.812409 24371 leader_election.cc:288] T d216a30d60604e44bea6d9a395508aa2 P 0083994f1c12460eb29337ad7b217e91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f704d805fef340d68c74e444508c238d (127.22.53.194:36981), 066cd56b67e94fc4932f012941b09759 (127.22.53.193:37391)
I20260926 07:29:50.817425 24359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:50.812757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.195:49787 (local address 127.22.53.194:36981)
0926 07:29:50.812934 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:50.812939 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:50.813138 (+ 199us) server_negotiation.cc:407] Connection header received
0926 07:29:50.813290 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:50.813294 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:50.813357 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:50.813446 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:29:50.813843 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:50.814500 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:50.816077 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:50.816732 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:50.816887 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:50.816947 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:50.817180 (+ 233us) server_negotiation.cc:299] Negotiation successful
0926 07:29:50.817293 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 07:29:50.817468 24362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:50.812903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.195:41705 (local address 127.22.53.193:37391)
0926 07:29:50.813042 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:50.813046 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:50.813283 (+ 237us) server_negotiation.cc:407] Connection header received
0926 07:29:50.813370 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:50.813373 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:50.813434 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:50.813507 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:29:50.814239 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:50.814852 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:50.816227 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:50.816942 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:50.817056 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:50.817092 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:50.817285 (+ 193us) server_negotiation.cc:299] Negotiation successful
0926 07:29:50.817360 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 07:29:50.817443 24234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:50.812778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.194:36981 (local address 127.22.53.195:49787)
0926 07:29:50.812969 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:50.812984 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:29:50.813138 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:29:50.813504 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:29:50.813507 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:29:50.813527 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:29:50.813699 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:50.813710 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:50.814661 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:50.814664 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:50.815935 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:50.815947 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:50.816852 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:50.816874 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:50.817020 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:50.817049 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 07:29:50.817141 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 07:29:50.817255 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"mutex_wait_us":39}
I20260926 07:29:50.817442 24373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:50.812808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.193:37391 (local address 127.22.53.195:41705)
0926 07:29:50.813181 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:50.813194 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:29:50.813286 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:29:50.813525 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:29:50.813528 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:29:50.813563 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:29:50.814099 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:50.814107 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:50.814976 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:50.814979 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:50.816119 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:50.816126 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:50.817047 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:50.817050 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:50.817154 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:50.817190 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 07:29:50.817255 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 07:29:50.817305 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"mutex_wait_us":4,"thread_start_us":149,"threads_started":1}
I20260926 07:29:50.818219 24145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d216a30d60604e44bea6d9a395508aa2" candidate_uuid: "0083994f1c12460eb29337ad7b217e91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f704d805fef340d68c74e444508c238d" is_pre_election: true
I20260926 07:29:50.818235 24015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d216a30d60604e44bea6d9a395508aa2" candidate_uuid: "0083994f1c12460eb29337ad7b217e91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "066cd56b67e94fc4932f012941b09759" is_pre_election: true
I20260926 07:29:50.818557 24145 raft_consensus.cc:2454] T d216a30d60604e44bea6d9a395508aa2 P f704d805fef340d68c74e444508c238d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0083994f1c12460eb29337ad7b217e91 in term 0.
I20260926 07:29:50.818562 24015 raft_consensus.cc:2454] T d216a30d60604e44bea6d9a395508aa2 P 066cd56b67e94fc4932f012941b09759 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0083994f1c12460eb29337ad7b217e91 in term 0.
I20260926 07:29:50.819119 24229 leader_election.cc:302] T d216a30d60604e44bea6d9a395508aa2 P 0083994f1c12460eb29337ad7b217e91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0083994f1c12460eb29337ad7b217e91, 066cd56b67e94fc4932f012941b09759, f704d805fef340d68c74e444508c238d; no voters:
I20260926 07:29:50.819469 24371 raft_consensus.cc:2781] T d216a30d60604e44bea6d9a395508aa2 P 0083994f1c12460eb29337ad7b217e91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:29:50.819591 24371 raft_consensus.cc:482] T d216a30d60604e44bea6d9a395508aa2 P 0083994f1c12460eb29337ad7b217e91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:29:50.819649 24371 raft_consensus.cc:3037] T d216a30d60604e44bea6d9a395508aa2 P 0083994f1c12460eb29337ad7b217e91 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:29:50.820835 24371 raft_consensus.cc:504] T d216a30d60604e44bea6d9a395508aa2 P 0083994f1c12460eb29337ad7b217e91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f704d805fef340d68c74e444508c238d" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 36981 } } peers { permanent_uuid: "066cd56b67e94fc4932f012941b09759" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 37391 } } peers { permanent_uuid: "0083994f1c12460eb29337ad7b217e91" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 45383 } }
I20260926 07:29:50.821281 24371 leader_election.cc:288] T d216a30d60604e44bea6d9a395508aa2 P 0083994f1c12460eb29337ad7b217e91 [CANDIDATE]: Term 1 election: Requested vote from peers f704d805fef340d68c74e444508c238d (127.22.53.194:36981), 066cd56b67e94fc4932f012941b09759 (127.22.53.193:37391)
I20260926 07:29:50.821727 24015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d216a30d60604e44bea6d9a395508aa2" candidate_uuid: "0083994f1c12460eb29337ad7b217e91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "066cd56b67e94fc4932f012941b09759"
I20260926 07:29:50.821727 24145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d216a30d60604e44bea6d9a395508aa2" candidate_uuid: "0083994f1c12460eb29337ad7b217e91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f704d805fef340d68c74e444508c238d"
I20260926 07:29:50.821877 24145 raft_consensus.cc:3037] T d216a30d60604e44bea6d9a395508aa2 P f704d805fef340d68c74e444508c238d [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:29:50.821877 24015 raft_consensus.cc:3037] T d216a30d60604e44bea6d9a395508aa2 P 066cd56b67e94fc4932f012941b09759 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:29:50.823232 24145 raft_consensus.cc:2454] T d216a30d60604e44bea6d9a395508aa2 P f704d805fef340d68c74e444508c238d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0083994f1c12460eb29337ad7b217e91 in term 1.
I20260926 07:29:50.823237 24015 raft_consensus.cc:2454] T d216a30d60604e44bea6d9a395508aa2 P 066cd56b67e94fc4932f012941b09759 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0083994f1c12460eb29337ad7b217e91 in term 1.
I20260926 07:29:50.823603 24229 leader_election.cc:302] T d216a30d60604e44bea6d9a395508aa2 P 0083994f1c12460eb29337ad7b217e91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0083994f1c12460eb29337ad7b217e91, 066cd56b67e94fc4932f012941b09759, f704d805fef340d68c74e444508c238d; no voters:
I20260926 07:29:50.823767 24371 raft_consensus.cc:2781] T d216a30d60604e44bea6d9a395508aa2 P 0083994f1c12460eb29337ad7b217e91 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:29:50.823997 24371 raft_consensus.cc:686] T d216a30d60604e44bea6d9a395508aa2 P 0083994f1c12460eb29337ad7b217e91 [term 1 LEADER]: Becoming Leader. State: Replica: 0083994f1c12460eb29337ad7b217e91, State: Running, Role: LEADER
I20260926 07:29:50.824378 24371 consensus_queue.cc:237] T d216a30d60604e44bea6d9a395508aa2 P 0083994f1c12460eb29337ad7b217e91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f704d805fef340d68c74e444508c238d" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 36981 } } peers { permanent_uuid: "066cd56b67e94fc4932f012941b09759" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 37391 } } peers { permanent_uuid: "0083994f1c12460eb29337ad7b217e91" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 45383 } }
I20260926 07:29:50.828186 23858 catalog_manager.cc:5495] T d216a30d60604e44bea6d9a395508aa2 P 0083994f1c12460eb29337ad7b217e91 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0083994f1c12460eb29337ad7b217e91 (127.22.53.195). New cstate: current_term: 1 leader_uuid: "0083994f1c12460eb29337ad7b217e91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f704d805fef340d68c74e444508c238d" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 36981 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "066cd56b67e94fc4932f012941b09759" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 37391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0083994f1c12460eb29337ad7b217e91" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 45383 } health_report { overall_health: HEALTHY } } }
I20260926 07:29:50.857162 24363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:50.853156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57140 (local address 127.22.53.195:45383)
0926 07:29:50.853333 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:50.853338 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:50.853355 (+ 17us) server_negotiation.cc:407] Connection header received
0926 07:29:50.853612 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:50.853615 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:50.853673 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:50.853792 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:29:50.854079 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:50.854596 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:50.855505 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:50.855740 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:50.855854 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:50.856229 (+ 375us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:29:50.856703 (+ 474us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:29:50.856831 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:50.856941 (+ 110us) server_negotiation.cc:299] Negotiation successful
0926 07:29:50.857002 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 07:29:50.863550 24145 raft_consensus.cc:1261] T d216a30d60604e44bea6d9a395508aa2 P f704d805fef340d68c74e444508c238d [term 1 FOLLOWER]: Refusing update from remote peer 0083994f1c12460eb29337ad7b217e91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:29:50.863551 24015 raft_consensus.cc:1261] T d216a30d60604e44bea6d9a395508aa2 P 066cd56b67e94fc4932f012941b09759 [term 1 FOLLOWER]: Refusing update from remote peer 0083994f1c12460eb29337ad7b217e91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:29:50.864357 24371 consensus_queue.cc:1035] T d216a30d60604e44bea6d9a395508aa2 P 0083994f1c12460eb29337ad7b217e91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "066cd56b67e94fc4932f012941b09759" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 37391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:29:50.864617 24374 consensus_queue.cc:1035] T d216a30d60604e44bea6d9a395508aa2 P 0083994f1c12460eb29337ad7b217e91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f704d805fef340d68c74e444508c238d" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 36981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:29:50.875641 24379 mvcc.cc:205] Tried to move back new op lower bound from 7333510311368916992 to 7333510311219515392. Current Snapshot: MvccSnapshot[applied={T|T < 7333510311368916992}]
I20260926 07:29:50.886148 23857 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58960:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:29:50.886379 23857 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58960:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:29:50.889364 23857 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7a0dca45b0b949f39066155a4f9f160f: Sending DeleteTablet for 3 replicas of tablet d216a30d60604e44bea6d9a395508aa2
I20260926 07:29:50.890369 24135 tablet_service.cc:1514] Processing DeleteTablet for tablet d216a30d60604e44bea6d9a395508aa2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:29:50 UTC) from {username='kudu', principal='kudu/127.22.53.254@KRBTEST.COM'} at 127.0.0.1:45356
I20260926 07:29:50.890450 23995 tablet_service.cc:1514] Processing DeleteTablet for tablet d216a30d60604e44bea6d9a395508aa2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:29:50 UTC) from {username='kudu', principal='kudu/127.22.53.254@KRBTEST.COM'} at 127.0.0.1:47934
I20260926 07:29:50.890954 24387 tablet_replica.cc:330] T d216a30d60604e44bea6d9a395508aa2 P f704d805fef340d68c74e444508c238d: stopping tablet replica
I20260926 07:29:50.891134 24275 tablet_service.cc:1514] Processing DeleteTablet for tablet d216a30d60604e44bea6d9a395508aa2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:29:50 UTC) from {username='kudu', principal='kudu/127.22.53.254@KRBTEST.COM'} at 127.0.0.1:57132
I20260926 07:29:50.891469 24387 raft_consensus.cc:2229] T d216a30d60604e44bea6d9a395508aa2 P f704d805fef340d68c74e444508c238d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:29:50.891671 24389 tablet_replica.cc:330] T d216a30d60604e44bea6d9a395508aa2 P 0083994f1c12460eb29337ad7b217e91: stopping tablet replica
I20260926 07:29:50.891948 24387 raft_consensus.cc:2258] T d216a30d60604e44bea6d9a395508aa2 P f704d805fef340d68c74e444508c238d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:29:50.891973 24389 raft_consensus.cc:2229] T d216a30d60604e44bea6d9a395508aa2 P 0083994f1c12460eb29337ad7b217e91 [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:29:50.891949 23857 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58960:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:29:50.892482 23857 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:29:50.892490 24389 raft_consensus.cc:2258] T d216a30d60604e44bea6d9a395508aa2 P 0083994f1c12460eb29337ad7b217e91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:29:50.893705 24388 tablet_replica.cc:330] T d216a30d60604e44bea6d9a395508aa2 P 066cd56b67e94fc4932f012941b09759: stopping tablet replica
I20260926 07:29:50.894033 24388 raft_consensus.cc:2229] T d216a30d60604e44bea6d9a395508aa2 P 066cd56b67e94fc4932f012941b09759 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:29:50.894367 24388 raft_consensus.cc:2258] T d216a30d60604e44bea6d9a395508aa2 P 066cd56b67e94fc4932f012941b09759 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:29:50.895275 24387 ts_tablet_manager.cc:1905] T d216a30d60604e44bea6d9a395508aa2 P f704d805fef340d68c74e444508c238d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:29:50.897004 24389 ts_tablet_manager.cc:1905] T d216a30d60604e44bea6d9a395508aa2 P 0083994f1c12460eb29337ad7b217e91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:29:50.898934 23995 tablet_service.cc:1467] Processing CreateTablet for tablet 52e070984444457bb145ad280db5926d (DEFAULT_TABLE table=test-table [id=5b8816b8199544de852b87c2426d4efa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:29:50.899021 24275 tablet_service.cc:1467] Processing CreateTablet for tablet 52e070984444457bb145ad280db5926d (DEFAULT_TABLE table=test-table [id=5b8816b8199544de852b87c2426d4efa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:29:50.899250 23995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52e070984444457bb145ad280db5926d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:29:50.899259 24275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52e070984444457bb145ad280db5926d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:29:50.899251 24135 tablet_service.cc:1467] Processing CreateTablet for tablet 52e070984444457bb145ad280db5926d (DEFAULT_TABLE table=test-table [id=5b8816b8199544de852b87c2426d4efa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:29:50.899526 24135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52e070984444457bb145ad280db5926d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:29:50.901865 24365 tablet_bootstrap.cc:492] T 52e070984444457bb145ad280db5926d P 066cd56b67e94fc4932f012941b09759: Bootstrap starting.
I20260926 07:29:50.902130 24364 tablet_bootstrap.cc:492] T 52e070984444457bb145ad280db5926d P 0083994f1c12460eb29337ad7b217e91: Bootstrap starting.
I20260926 07:29:50.902845 24365 tablet_bootstrap.cc:654] T 52e070984444457bb145ad280db5926d P 066cd56b67e94fc4932f012941b09759: Neither blocks nor log segments found. Creating new log.
I20260926 07:29:50.903141 24364 tablet_bootstrap.cc:654] T 52e070984444457bb145ad280db5926d P 0083994f1c12460eb29337ad7b217e91: Neither blocks nor log segments found. Creating new log.
I20260926 07:29:50.903790 24366 tablet_bootstrap.cc:492] T 52e070984444457bb145ad280db5926d P f704d805fef340d68c74e444508c238d: Bootstrap starting.
I20260926 07:29:50.904254 24365 tablet_bootstrap.cc:492] T 52e070984444457bb145ad280db5926d P 066cd56b67e94fc4932f012941b09759: No bootstrap required, opened a new log
I20260926 07:29:50.904387 24365 ts_tablet_manager.cc:1397] T 52e070984444457bb145ad280db5926d P 066cd56b67e94fc4932f012941b09759: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 07:29:50.904844 24366 tablet_bootstrap.cc:654] T 52e070984444457bb145ad280db5926d P f704d805fef340d68c74e444508c238d: Neither blocks nor log segments found. Creating new log.
I20260926 07:29:50.904858 24365 raft_consensus.cc:348] T 52e070984444457bb145ad280db5926d P 066cd56b67e94fc4932f012941b09759 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "066cd56b67e94fc4932f012941b09759" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 37391 } } peers { permanent_uuid: "f704d805fef340d68c74e444508c238d" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 36981 } } peers { permanent_uuid: "0083994f1c12460eb29337ad7b217e91" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 45383 } }
I20260926 07:29:50.905009 24365 raft_consensus.cc:374] T 52e070984444457bb145ad280db5926d P 066cd56b67e94fc4932f012941b09759 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:29:50.905048 24365 raft_consensus.cc:729] T 52e070984444457bb145ad280db5926d P 066cd56b67e94fc4932f012941b09759 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 066cd56b67e94fc4932f012941b09759, State: Initialized, Role: FOLLOWER
I20260926 07:29:50.905227 24365 consensus_queue.cc:260] T 52e070984444457bb145ad280db5926d P 066cd56b67e94fc4932f012941b09759 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "066cd56b67e94fc4932f012941b09759" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 37391 } } peers { permanent_uuid: "f704d805fef340d68c74e444508c238d" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 36981 } } peers { permanent_uuid: "0083994f1c12460eb29337ad7b217e91" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 45383 } }
I20260926 07:29:50.905606 24365 ts_tablet_manager.cc:1428] T 52e070984444457bb145ad280db5926d P 066cd56b67e94fc4932f012941b09759: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260926 07:29:50.898630 24388 ts_tablet_manager.cc:1905] T d216a30d60604e44bea6d9a395508aa2 P 066cd56b67e94fc4932f012941b09759: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:29:50.909117 24388 ts_tablet_manager.cc:1918] T d216a30d60604e44bea6d9a395508aa2 P 066cd56b67e94fc4932f012941b09759: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:29:50.909251 24388 log.cc:1196] T d216a30d60604e44bea6d9a395508aa2 P 066cd56b67e94fc4932f012941b09759: Deleting WAL directory at /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407783529717-22743-0/minicluster-data/ts-0/wal/wals/d216a30d60604e44bea6d9a395508aa2
I20260926 07:29:50.909600 24388 ts_tablet_manager.cc:1939] T d216a30d60604e44bea6d9a395508aa2 P 066cd56b67e94fc4932f012941b09759: Deleting consensus metadata
I20260926 07:29:50.910897 24389 ts_tablet_manager.cc:1918] T d216a30d60604e44bea6d9a395508aa2 P 0083994f1c12460eb29337ad7b217e91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:29:50.910943 23845 catalog_manager.cc:4841] TS 066cd56b67e94fc4932f012941b09759 (127.22.53.193:37391): tablet d216a30d60604e44bea6d9a395508aa2 (table test-table [id=aad6677a503e4c5b95908776ed0385b7]) successfully deleted
I20260926 07:29:50.911012 24389 log.cc:1196] T d216a30d60604e44bea6d9a395508aa2 P 0083994f1c12460eb29337ad7b217e91: Deleting WAL directory at /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407783529717-22743-0/minicluster-data/ts-2/wal/wals/d216a30d60604e44bea6d9a395508aa2
I20260926 07:29:50.911347 24389 ts_tablet_manager.cc:1939] T d216a30d60604e44bea6d9a395508aa2 P 0083994f1c12460eb29337ad7b217e91: Deleting consensus metadata
I20260926 07:29:50.912705 24387 ts_tablet_manager.cc:1918] T d216a30d60604e44bea6d9a395508aa2 P f704d805fef340d68c74e444508c238d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:29:50.912763 24364 tablet_bootstrap.cc:492] T 52e070984444457bb145ad280db5926d P 0083994f1c12460eb29337ad7b217e91: No bootstrap required, opened a new log
I20260926 07:29:50.912855 24387 log.cc:1196] T d216a30d60604e44bea6d9a395508aa2 P f704d805fef340d68c74e444508c238d: Deleting WAL directory at /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407783529717-22743-0/minicluster-data/ts-1/wal/wals/d216a30d60604e44bea6d9a395508aa2
I20260926 07:29:50.912861 24364 ts_tablet_manager.cc:1397] T 52e070984444457bb145ad280db5926d P 0083994f1c12460eb29337ad7b217e91: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20260926 07:29:50.913215 24387 ts_tablet_manager.cc:1939] T d216a30d60604e44bea6d9a395508aa2 P f704d805fef340d68c74e444508c238d: Deleting consensus metadata
I20260926 07:29:50.913383 24364 raft_consensus.cc:348] T 52e070984444457bb145ad280db5926d P 0083994f1c12460eb29337ad7b217e91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "066cd56b67e94fc4932f012941b09759" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 37391 } } peers { permanent_uuid: "f704d805fef340d68c74e444508c238d" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 36981 } } peers { permanent_uuid: "0083994f1c12460eb29337ad7b217e91" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 45383 } }
I20260926 07:29:50.913516 24364 raft_consensus.cc:374] T 52e070984444457bb145ad280db5926d P 0083994f1c12460eb29337ad7b217e91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:29:50.913546 24364 raft_consensus.cc:729] T 52e070984444457bb145ad280db5926d P 0083994f1c12460eb29337ad7b217e91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0083994f1c12460eb29337ad7b217e91, State: Initialized, Role: FOLLOWER
I20260926 07:29:50.913676 24364 consensus_queue.cc:260] T 52e070984444457bb145ad280db5926d P 0083994f1c12460eb29337ad7b217e91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "066cd56b67e94fc4932f012941b09759" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 37391 } } peers { permanent_uuid: "f704d805fef340d68c74e444508c238d" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 36981 } } peers { permanent_uuid: "0083994f1c12460eb29337ad7b217e91" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 45383 } }
I20260926 07:29:50.913934 24364 ts_tablet_manager.cc:1428] T 52e070984444457bb145ad280db5926d P 0083994f1c12460eb29337ad7b217e91: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 07:29:50.914712 23844 catalog_manager.cc:4841] TS f704d805fef340d68c74e444508c238d (127.22.53.194:36981): tablet d216a30d60604e44bea6d9a395508aa2 (table test-table [id=aad6677a503e4c5b95908776ed0385b7]) successfully deleted
I20260926 07:29:50.915387 24366 tablet_bootstrap.cc:492] T 52e070984444457bb145ad280db5926d P f704d805fef340d68c74e444508c238d: No bootstrap required, opened a new log
I20260926 07:29:50.915498 24366 ts_tablet_manager.cc:1397] T 52e070984444457bb145ad280db5926d P f704d805fef340d68c74e444508c238d: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20260926 07:29:50.915638 23845 catalog_manager.cc:4841] TS 0083994f1c12460eb29337ad7b217e91 (127.22.53.195:45383): tablet d216a30d60604e44bea6d9a395508aa2 (table test-table [id=aad6677a503e4c5b95908776ed0385b7]) successfully deleted
I20260926 07:29:50.915983 24366 raft_consensus.cc:348] T 52e070984444457bb145ad280db5926d P f704d805fef340d68c74e444508c238d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "066cd56b67e94fc4932f012941b09759" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 37391 } } peers { permanent_uuid: "f704d805fef340d68c74e444508c238d" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 36981 } } peers { permanent_uuid: "0083994f1c12460eb29337ad7b217e91" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 45383 } }
I20260926 07:29:50.916224 24366 raft_consensus.cc:374] T 52e070984444457bb145ad280db5926d P f704d805fef340d68c74e444508c238d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:29:50.916304 24366 raft_consensus.cc:729] T 52e070984444457bb145ad280db5926d P f704d805fef340d68c74e444508c238d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f704d805fef340d68c74e444508c238d, State: Initialized, Role: FOLLOWER
I20260926 07:29:50.916457 24366 consensus_queue.cc:260] T 52e070984444457bb145ad280db5926d P f704d805fef340d68c74e444508c238d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "066cd56b67e94fc4932f012941b09759" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 37391 } } peers { permanent_uuid: "f704d805fef340d68c74e444508c238d" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 36981 } } peers { permanent_uuid: "0083994f1c12460eb29337ad7b217e91" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 45383 } }
I20260926 07:29:50.916826 24366 ts_tablet_manager.cc:1428] T 52e070984444457bb145ad280db5926d P f704d805fef340d68c74e444508c238d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 07:29:51.091958 24384 raft_consensus.cc:482] T 52e070984444457bb145ad280db5926d P 0083994f1c12460eb29337ad7b217e91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:29:51.092140 24384 raft_consensus.cc:504] T 52e070984444457bb145ad280db5926d P 0083994f1c12460eb29337ad7b217e91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "066cd56b67e94fc4932f012941b09759" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 37391 } } peers { permanent_uuid: "f704d805fef340d68c74e444508c238d" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 36981 } } peers { permanent_uuid: "0083994f1c12460eb29337ad7b217e91" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 45383 } }
I20260926 07:29:51.092713 24384 leader_election.cc:288] T 52e070984444457bb145ad280db5926d P 0083994f1c12460eb29337ad7b217e91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 066cd56b67e94fc4932f012941b09759 (127.22.53.193:37391), f704d805fef340d68c74e444508c238d (127.22.53.194:36981)
I20260926 07:29:51.093313 24015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52e070984444457bb145ad280db5926d" candidate_uuid: "0083994f1c12460eb29337ad7b217e91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "066cd56b67e94fc4932f012941b09759" is_pre_election: true
I20260926 07:29:51.093323 24145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52e070984444457bb145ad280db5926d" candidate_uuid: "0083994f1c12460eb29337ad7b217e91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f704d805fef340d68c74e444508c238d" is_pre_election: true
I20260926 07:29:51.093461 24015 raft_consensus.cc:2454] T 52e070984444457bb145ad280db5926d P 066cd56b67e94fc4932f012941b09759 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0083994f1c12460eb29337ad7b217e91 in term 0.
I20260926 07:29:51.093520 24145 raft_consensus.cc:2454] T 52e070984444457bb145ad280db5926d P f704d805fef340d68c74e444508c238d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0083994f1c12460eb29337ad7b217e91 in term 0.
I20260926 07:29:51.093849 24229 leader_election.cc:302] T 52e070984444457bb145ad280db5926d P 0083994f1c12460eb29337ad7b217e91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0083994f1c12460eb29337ad7b217e91, 066cd56b67e94fc4932f012941b09759; no voters:
I20260926 07:29:51.094076 24384 raft_consensus.cc:2781] T 52e070984444457bb145ad280db5926d P 0083994f1c12460eb29337ad7b217e91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:29:51.094149 24384 raft_consensus.cc:482] T 52e070984444457bb145ad280db5926d P 0083994f1c12460eb29337ad7b217e91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:29:51.094192 24384 raft_consensus.cc:3037] T 52e070984444457bb145ad280db5926d P 0083994f1c12460eb29337ad7b217e91 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:29:51.095414 24384 raft_consensus.cc:504] T 52e070984444457bb145ad280db5926d P 0083994f1c12460eb29337ad7b217e91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "066cd56b67e94fc4932f012941b09759" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 37391 } } peers { permanent_uuid: "f704d805fef340d68c74e444508c238d" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 36981 } } peers { permanent_uuid: "0083994f1c12460eb29337ad7b217e91" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 45383 } }
I20260926 07:29:51.095803 24384 leader_election.cc:288] T 52e070984444457bb145ad280db5926d P 0083994f1c12460eb29337ad7b217e91 [CANDIDATE]: Term 1 election: Requested vote from peers 066cd56b67e94fc4932f012941b09759 (127.22.53.193:37391), f704d805fef340d68c74e444508c238d (127.22.53.194:36981)
I20260926 07:29:51.096285 24015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52e070984444457bb145ad280db5926d" candidate_uuid: "0083994f1c12460eb29337ad7b217e91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "066cd56b67e94fc4932f012941b09759"
I20260926 07:29:51.096294 24145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52e070984444457bb145ad280db5926d" candidate_uuid: "0083994f1c12460eb29337ad7b217e91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f704d805fef340d68c74e444508c238d"
I20260926 07:29:51.096437 24145 raft_consensus.cc:3037] T 52e070984444457bb145ad280db5926d P f704d805fef340d68c74e444508c238d [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:29:51.096437 24015 raft_consensus.cc:3037] T 52e070984444457bb145ad280db5926d P 066cd56b67e94fc4932f012941b09759 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:29:51.097643 24145 raft_consensus.cc:2454] T 52e070984444457bb145ad280db5926d P f704d805fef340d68c74e444508c238d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0083994f1c12460eb29337ad7b217e91 in term 1.
I20260926 07:29:51.097644 24015 raft_consensus.cc:2454] T 52e070984444457bb145ad280db5926d P 066cd56b67e94fc4932f012941b09759 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0083994f1c12460eb29337ad7b217e91 in term 1.
I20260926 07:29:51.098028 24229 leader_election.cc:302] T 52e070984444457bb145ad280db5926d P 0083994f1c12460eb29337ad7b217e91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0083994f1c12460eb29337ad7b217e91, 066cd56b67e94fc4932f012941b09759, f704d805fef340d68c74e444508c238d; no voters:
I20260926 07:29:51.098264 24384 raft_consensus.cc:2781] T 52e070984444457bb145ad280db5926d P 0083994f1c12460eb29337ad7b217e91 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:29:51.098385 24384 raft_consensus.cc:686] T 52e070984444457bb145ad280db5926d P 0083994f1c12460eb29337ad7b217e91 [term 1 LEADER]: Becoming Leader. State: Replica: 0083994f1c12460eb29337ad7b217e91, State: Running, Role: LEADER
I20260926 07:29:51.098557 24384 consensus_queue.cc:237] T 52e070984444457bb145ad280db5926d P 0083994f1c12460eb29337ad7b217e91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "066cd56b67e94fc4932f012941b09759" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 37391 } } peers { permanent_uuid: "f704d805fef340d68c74e444508c238d" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 36981 } } peers { permanent_uuid: "0083994f1c12460eb29337ad7b217e91" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 45383 } }
I20260926 07:29:51.100481 23858 catalog_manager.cc:5495] T 52e070984444457bb145ad280db5926d P 0083994f1c12460eb29337ad7b217e91 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0083994f1c12460eb29337ad7b217e91 (127.22.53.195). New cstate: current_term: 1 leader_uuid: "0083994f1c12460eb29337ad7b217e91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "066cd56b67e94fc4932f012941b09759" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 37391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f704d805fef340d68c74e444508c238d" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 36981 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0083994f1c12460eb29337ad7b217e91" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 45383 } health_report { overall_health: HEALTHY } } }
Sep 26 07:29:51 dist-test-slave-wdjc krb5kdc[23813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407789, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.254@KRBTEST.COM for kudu/127.22.53.254@KRBTEST.COM
I20260926 07:29:51.180207 24399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:51.170517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:44201 (local address 127.0.0.1:58972)
0926 07:29:51.170871 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:51.170885 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:29:51.170990 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:29:51.171266 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:29:51.171268 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:29:51.171687 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:29:51.171795 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:51.171801 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:51.172719 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:51.172722 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:51.173321 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:51.173330 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:51.173834 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:51.173837 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:51.173906 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:51.174551 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:29:51.174564 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:29:51.176306 (+ 1742us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:29:51.178401 (+ 2095us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:29:51.178415 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:29:51.178420 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:29:51.178695 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:29:51.179075 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:29:51.179077 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:29:51.179079 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:29:51.179171 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:29:51.179650 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:29:51.179657 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:29:51.179781 (+ 124us) client_negotiation.cc:769] Sending connection context
0926 07:29:51.179896 (+ 115us) client_negotiation.cc:240] Negotiation successful
0926 07:29:51.180052 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":125,"threads_started":1}
I20260926 07:29:51.180299 24400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:51.170620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58972 (local address 127.22.53.254:44201)
0926 07:29:51.170953 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:51.170960 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:51.170975 (+ 15us) server_negotiation.cc:407] Connection header received
0926 07:29:51.171088 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:51.171090 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:51.171132 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:51.171214 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:51.171952 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:51.172507 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:51.173448 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:51.173690 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:51.173799 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:51.176477 (+ 2678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:51.176480 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:51.176482 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:29:51.176506 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:51.178250 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:51.178825 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:51.178828 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:51.178829 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:51.178925 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:51.179272 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:51.179275 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:51.179276 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:51.179484 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:51.179569 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:51.179960 (+ 391us) server_negotiation.cc:299] Negotiation successful
0926 07:29:51.180142 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":273,"thread_start_us":142,"threads_started":1}
I20260926 07:29:51.184893 23858 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.22.53.254@KRBTEST.COM'} at 127.0.0.1:58972:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 07:29:51.185719 23858 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:29:51.191191 24135 tablet_service.cc:1467] Processing CreateTablet for tablet c874cfc77fe64dfe8c8321aa3a09138e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=17538632dfb749998ddc3623b575d247]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 07:29:51.191191 23995 tablet_service.cc:1467] Processing CreateTablet for tablet c874cfc77fe64dfe8c8321aa3a09138e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=17538632dfb749998ddc3623b575d247]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 07:29:51.191253 24275 tablet_service.cc:1467] Processing CreateTablet for tablet c874cfc77fe64dfe8c8321aa3a09138e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=17538632dfb749998ddc3623b575d247]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 07:29:51.191632 23995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c874cfc77fe64dfe8c8321aa3a09138e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:29:51.191632 24135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c874cfc77fe64dfe8c8321aa3a09138e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:29:51.191668 24275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c874cfc77fe64dfe8c8321aa3a09138e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:29:51.194617 24364 tablet_bootstrap.cc:492] T c874cfc77fe64dfe8c8321aa3a09138e P 0083994f1c12460eb29337ad7b217e91: Bootstrap starting.
I20260926 07:29:51.194801 24366 tablet_bootstrap.cc:492] T c874cfc77fe64dfe8c8321aa3a09138e P f704d805fef340d68c74e444508c238d: Bootstrap starting.
I20260926 07:29:51.195268 24365 tablet_bootstrap.cc:492] T c874cfc77fe64dfe8c8321aa3a09138e P 066cd56b67e94fc4932f012941b09759: Bootstrap starting.
I20260926 07:29:51.195675 24364 tablet_bootstrap.cc:654] T c874cfc77fe64dfe8c8321aa3a09138e P 0083994f1c12460eb29337ad7b217e91: Neither blocks nor log segments found. Creating new log.
I20260926 07:29:51.195771 24366 tablet_bootstrap.cc:654] T c874cfc77fe64dfe8c8321aa3a09138e P f704d805fef340d68c74e444508c238d: Neither blocks nor log segments found. Creating new log.
I20260926 07:29:51.196012 24365 tablet_bootstrap.cc:654] T c874cfc77fe64dfe8c8321aa3a09138e P 066cd56b67e94fc4932f012941b09759: Neither blocks nor log segments found. Creating new log.
I20260926 07:29:51.197075 24364 tablet_bootstrap.cc:492] T c874cfc77fe64dfe8c8321aa3a09138e P 0083994f1c12460eb29337ad7b217e91: No bootstrap required, opened a new log
I20260926 07:29:51.197185 24364 ts_tablet_manager.cc:1397] T c874cfc77fe64dfe8c8321aa3a09138e P 0083994f1c12460eb29337ad7b217e91: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 07:29:51.197221 24365 tablet_bootstrap.cc:492] T c874cfc77fe64dfe8c8321aa3a09138e P 066cd56b67e94fc4932f012941b09759: No bootstrap required, opened a new log
I20260926 07:29:51.197275 24366 tablet_bootstrap.cc:492] T c874cfc77fe64dfe8c8321aa3a09138e P f704d805fef340d68c74e444508c238d: No bootstrap required, opened a new log
I20260926 07:29:51.197306 24365 ts_tablet_manager.cc:1397] T c874cfc77fe64dfe8c8321aa3a09138e P 066cd56b67e94fc4932f012941b09759: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 07:29:51.197376 24366 ts_tablet_manager.cc:1397] T c874cfc77fe64dfe8c8321aa3a09138e P f704d805fef340d68c74e444508c238d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 07:29:51.197714 24365 raft_consensus.cc:348] T c874cfc77fe64dfe8c8321aa3a09138e P 066cd56b67e94fc4932f012941b09759 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "066cd56b67e94fc4932f012941b09759" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 37391 } } peers { permanent_uuid: "f704d805fef340d68c74e444508c238d" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 36981 } } peers { permanent_uuid: "0083994f1c12460eb29337ad7b217e91" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 45383 } }
I20260926 07:29:51.197672 24364 raft_consensus.cc:348] T c874cfc77fe64dfe8c8321aa3a09138e P 0083994f1c12460eb29337ad7b217e91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "066cd56b67e94fc4932f012941b09759" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 37391 } } peers { permanent_uuid: "f704d805fef340d68c74e444508c238d" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 36981 } } peers { permanent_uuid: "0083994f1c12460eb29337ad7b217e91" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 45383 } }
I20260926 07:29:51.197826 24364 raft_consensus.cc:374] T c874cfc77fe64dfe8c8321aa3a09138e P 0083994f1c12460eb29337ad7b217e91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:29:51.197826 24365 raft_consensus.cc:374] T c874cfc77fe64dfe8c8321aa3a09138e P 066cd56b67e94fc4932f012941b09759 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:29:51.197863 24365 raft_consensus.cc:729] T c874cfc77fe64dfe8c8321aa3a09138e P 066cd56b67e94fc4932f012941b09759 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 066cd56b67e94fc4932f012941b09759, State: Initialized, Role: FOLLOWER
I20260926 07:29:51.197863 24364 raft_consensus.cc:729] T c874cfc77fe64dfe8c8321aa3a09138e P 0083994f1c12460eb29337ad7b217e91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0083994f1c12460eb29337ad7b217e91, State: Initialized, Role: FOLLOWER
I20260926 07:29:51.197788 24366 raft_consensus.cc:348] T c874cfc77fe64dfe8c8321aa3a09138e P f704d805fef340d68c74e444508c238d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "066cd56b67e94fc4932f012941b09759" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 37391 } } peers { permanent_uuid: "f704d805fef340d68c74e444508c238d" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 36981 } } peers { permanent_uuid: "0083994f1c12460eb29337ad7b217e91" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 45383 } }
I20260926 07:29:51.197929 24366 raft_consensus.cc:374] T c874cfc77fe64dfe8c8321aa3a09138e P f704d805fef340d68c74e444508c238d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:29:51.197970 24366 raft_consensus.cc:729] T c874cfc77fe64dfe8c8321aa3a09138e P f704d805fef340d68c74e444508c238d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f704d805fef340d68c74e444508c238d, State: Initialized, Role: FOLLOWER
I20260926 07:29:51.197997 24364 consensus_queue.cc:260] T c874cfc77fe64dfe8c8321aa3a09138e P 0083994f1c12460eb29337ad7b217e91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "066cd56b67e94fc4932f012941b09759" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 37391 } } peers { permanent_uuid: "f704d805fef340d68c74e444508c238d" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 36981 } } peers { permanent_uuid: "0083994f1c12460eb29337ad7b217e91" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 45383 } }
I20260926 07:29:51.197997 24365 consensus_queue.cc:260] T c874cfc77fe64dfe8c8321aa3a09138e P 066cd56b67e94fc4932f012941b09759 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "066cd56b67e94fc4932f012941b09759" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 37391 } } peers { permanent_uuid: "f704d805fef340d68c74e444508c238d" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 36981 } } peers { permanent_uuid: "0083994f1c12460eb29337ad7b217e91" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 45383 } }
I20260926 07:29:51.198117 24366 consensus_queue.cc:260] T c874cfc77fe64dfe8c8321aa3a09138e P f704d805fef340d68c74e444508c238d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "066cd56b67e94fc4932f012941b09759" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 37391 } } peers { permanent_uuid: "f704d805fef340d68c74e444508c238d" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 36981 } } peers { permanent_uuid: "0083994f1c12460eb29337ad7b217e91" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 45383 } }
I20260926 07:29:51.198298 24364 ts_tablet_manager.cc:1428] T c874cfc77fe64dfe8c8321aa3a09138e P 0083994f1c12460eb29337ad7b217e91: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 07:29:51.198307 24365 ts_tablet_manager.cc:1428] T c874cfc77fe64dfe8c8321aa3a09138e P 066cd56b67e94fc4932f012941b09759: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260926 07:29:51.198486 24366 ts_tablet_manager.cc:1428] T c874cfc77fe64dfe8c8321aa3a09138e P f704d805fef340d68c74e444508c238d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 07:29:51.198455 24372 tablet_replica.cc:439] T c874cfc77fe64dfe8c8321aa3a09138e P f704d805fef340d68c74e444508c238d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "066cd56b67e94fc4932f012941b09759" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 37391 } } peers { permanent_uuid: "f704d805fef340d68c74e444508c238d" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 36981 } } peers { permanent_uuid: "0083994f1c12460eb29337ad7b217e91" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 45383 } } }
I20260926 07:29:51.198609 24372 tablet_replica.cc:442] T c874cfc77fe64dfe8c8321aa3a09138e P f704d805fef340d68c74e444508c238d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:29:51.198544 24384 tablet_replica.cc:439] T c874cfc77fe64dfe8c8321aa3a09138e P 0083994f1c12460eb29337ad7b217e91: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "066cd56b67e94fc4932f012941b09759" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 37391 } } peers { permanent_uuid: "f704d805fef340d68c74e444508c238d" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 36981 } } peers { permanent_uuid: "0083994f1c12460eb29337ad7b217e91" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 45383 } } }
I20260926 07:29:51.198675 24384 tablet_replica.cc:442] T c874cfc77fe64dfe8c8321aa3a09138e P 0083994f1c12460eb29337ad7b217e91: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:29:51.198879 24370 tablet_replica.cc:439] T c874cfc77fe64dfe8c8321aa3a09138e P 066cd56b67e94fc4932f012941b09759: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "066cd56b67e94fc4932f012941b09759" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 37391 } } peers { permanent_uuid: "f704d805fef340d68c74e444508c238d" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 36981 } } peers { permanent_uuid: "0083994f1c12460eb29337ad7b217e91" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 45383 } } }
I20260926 07:29:51.199018 24370 tablet_replica.cc:442] T c874cfc77fe64dfe8c8321aa3a09138e P 066cd56b67e94fc4932f012941b09759: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:29:51.480996 24384 raft_consensus.cc:482] T c874cfc77fe64dfe8c8321aa3a09138e P 0083994f1c12460eb29337ad7b217e91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:29:51.481194 24384 raft_consensus.cc:504] T c874cfc77fe64dfe8c8321aa3a09138e P 0083994f1c12460eb29337ad7b217e91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "066cd56b67e94fc4932f012941b09759" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 37391 } } peers { permanent_uuid: "f704d805fef340d68c74e444508c238d" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 36981 } } peers { permanent_uuid: "0083994f1c12460eb29337ad7b217e91" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 45383 } }
I20260926 07:29:51.481705 24384 leader_election.cc:288] T c874cfc77fe64dfe8c8321aa3a09138e P 0083994f1c12460eb29337ad7b217e91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 066cd56b67e94fc4932f012941b09759 (127.22.53.193:37391), f704d805fef340d68c74e444508c238d (127.22.53.194:36981)
I20260926 07:29:51.482285 24145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c874cfc77fe64dfe8c8321aa3a09138e" candidate_uuid: "0083994f1c12460eb29337ad7b217e91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f704d805fef340d68c74e444508c238d" is_pre_election: true
I20260926 07:29:51.482286 24015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c874cfc77fe64dfe8c8321aa3a09138e" candidate_uuid: "0083994f1c12460eb29337ad7b217e91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "066cd56b67e94fc4932f012941b09759" is_pre_election: true
I20260926 07:29:51.482466 24145 raft_consensus.cc:2454] T c874cfc77fe64dfe8c8321aa3a09138e P f704d805fef340d68c74e444508c238d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0083994f1c12460eb29337ad7b217e91 in term 0.
I20260926 07:29:51.482475 24015 raft_consensus.cc:2454] T c874cfc77fe64dfe8c8321aa3a09138e P 066cd56b67e94fc4932f012941b09759 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0083994f1c12460eb29337ad7b217e91 in term 0.
I20260926 07:29:51.482853 24229 leader_election.cc:302] T c874cfc77fe64dfe8c8321aa3a09138e P 0083994f1c12460eb29337ad7b217e91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0083994f1c12460eb29337ad7b217e91, 066cd56b67e94fc4932f012941b09759; no voters:
I20260926 07:29:51.483059 24384 raft_consensus.cc:2781] T c874cfc77fe64dfe8c8321aa3a09138e P 0083994f1c12460eb29337ad7b217e91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:29:51.483139 24384 raft_consensus.cc:482] T c874cfc77fe64dfe8c8321aa3a09138e P 0083994f1c12460eb29337ad7b217e91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:29:51.483201 24384 raft_consensus.cc:3037] T c874cfc77fe64dfe8c8321aa3a09138e P 0083994f1c12460eb29337ad7b217e91 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:29:51.484267 24384 raft_consensus.cc:504] T c874cfc77fe64dfe8c8321aa3a09138e P 0083994f1c12460eb29337ad7b217e91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "066cd56b67e94fc4932f012941b09759" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 37391 } } peers { permanent_uuid: "f704d805fef340d68c74e444508c238d" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 36981 } } peers { permanent_uuid: "0083994f1c12460eb29337ad7b217e91" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 45383 } }
I20260926 07:29:51.484817 24384 leader_election.cc:288] T c874cfc77fe64dfe8c8321aa3a09138e P 0083994f1c12460eb29337ad7b217e91 [CANDIDATE]: Term 1 election: Requested vote from peers 066cd56b67e94fc4932f012941b09759 (127.22.53.193:37391), f704d805fef340d68c74e444508c238d (127.22.53.194:36981)
I20260926 07:29:51.485256 24015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c874cfc77fe64dfe8c8321aa3a09138e" candidate_uuid: "0083994f1c12460eb29337ad7b217e91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "066cd56b67e94fc4932f012941b09759"
I20260926 07:29:51.485256 24145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c874cfc77fe64dfe8c8321aa3a09138e" candidate_uuid: "0083994f1c12460eb29337ad7b217e91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f704d805fef340d68c74e444508c238d"
I20260926 07:29:51.485397 24015 raft_consensus.cc:3037] T c874cfc77fe64dfe8c8321aa3a09138e P 066cd56b67e94fc4932f012941b09759 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:29:51.485397 24145 raft_consensus.cc:3037] T c874cfc77fe64dfe8c8321aa3a09138e P f704d805fef340d68c74e444508c238d [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:29:51.486604 24145 raft_consensus.cc:2454] T c874cfc77fe64dfe8c8321aa3a09138e P f704d805fef340d68c74e444508c238d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0083994f1c12460eb29337ad7b217e91 in term 1.
I20260926 07:29:51.486605 24015 raft_consensus.cc:2454] T c874cfc77fe64dfe8c8321aa3a09138e P 066cd56b67e94fc4932f012941b09759 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0083994f1c12460eb29337ad7b217e91 in term 1.
I20260926 07:29:51.486987 24228 leader_election.cc:302] T c874cfc77fe64dfe8c8321aa3a09138e P 0083994f1c12460eb29337ad7b217e91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0083994f1c12460eb29337ad7b217e91, f704d805fef340d68c74e444508c238d; no voters:
I20260926 07:29:51.487201 24384 raft_consensus.cc:2781] T c874cfc77fe64dfe8c8321aa3a09138e P 0083994f1c12460eb29337ad7b217e91 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:29:51.487295 24384 raft_consensus.cc:686] T c874cfc77fe64dfe8c8321aa3a09138e P 0083994f1c12460eb29337ad7b217e91 [term 1 LEADER]: Becoming Leader. State: Replica: 0083994f1c12460eb29337ad7b217e91, State: Running, Role: LEADER
I20260926 07:29:51.487450 24384 consensus_queue.cc:237] T c874cfc77fe64dfe8c8321aa3a09138e P 0083994f1c12460eb29337ad7b217e91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "066cd56b67e94fc4932f012941b09759" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 37391 } } peers { permanent_uuid: "f704d805fef340d68c74e444508c238d" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 36981 } } peers { permanent_uuid: "0083994f1c12460eb29337ad7b217e91" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 45383 } }
I20260926 07:29:51.488183 24374 tablet_replica.cc:439] T c874cfc77fe64dfe8c8321aa3a09138e P 0083994f1c12460eb29337ad7b217e91: TxnStatusTablet state changed. Reason: New leader 0083994f1c12460eb29337ad7b217e91. Latest consensus state: current_term: 1 leader_uuid: "0083994f1c12460eb29337ad7b217e91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "066cd56b67e94fc4932f012941b09759" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 37391 } } peers { permanent_uuid: "f704d805fef340d68c74e444508c238d" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 36981 } } peers { permanent_uuid: "0083994f1c12460eb29337ad7b217e91" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 45383 } } }
I20260926 07:29:51.488313 24374 tablet_replica.cc:442] T c874cfc77fe64dfe8c8321aa3a09138e P 0083994f1c12460eb29337ad7b217e91: This TxnStatusTablet replica's current role is: LEADER
I20260926 07:29:51.488517 24405 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 07:29:51.488729 24405 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 07:29:51.489331 23856 catalog_manager.cc:5495] T c874cfc77fe64dfe8c8321aa3a09138e P 0083994f1c12460eb29337ad7b217e91 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0083994f1c12460eb29337ad7b217e91 (127.22.53.195). New cstate: current_term: 1 leader_uuid: "0083994f1c12460eb29337ad7b217e91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "066cd56b67e94fc4932f012941b09759" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 37391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f704d805fef340d68c74e444508c238d" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 36981 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0083994f1c12460eb29337ad7b217e91" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 45383 } health_report { overall_health: HEALTHY } } }
I20260926 07:29:51.569507 24408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:51.565766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57150 (local address 127.22.53.195:45383)
0926 07:29:51.566220 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:51.566224 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:51.566247 (+ 23us) server_negotiation.cc:407] Connection header received
0926 07:29:51.566297 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:51.566300 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:51.566345 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:51.566453 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:29:51.566753 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:51.567280 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:51.568184 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:51.568426 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:51.568539 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:51.568921 (+ 382us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:29:51.569015 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:29:51.569103 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:51.569248 (+ 145us) server_negotiation.cc:299] Negotiation successful
0926 07:29:51.569319 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":99,"threads_started":1}
I20260926 07:29:51.569483 24399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:51.565599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.195:45383 (local address 127.0.0.1:57150)
0926 07:29:51.565831 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:51.565853 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 07:29:51.565998 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:29:51.566480 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:29:51.566483 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:29:51.566491 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 07:29:51.566617 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:51.566623 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:51.567406 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:51.567410 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:51.568041 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:51.568051 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:51.568587 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:51.568592 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:51.568689 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:51.568756 (+ 67us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 07:29:51.569108 (+ 352us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 07:29:51.569130 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 07:29:51.569223 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 07:29:51.569337 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88}
I20260926 07:29:51.572796 24145 raft_consensus.cc:1261] T c874cfc77fe64dfe8c8321aa3a09138e P f704d805fef340d68c74e444508c238d [term 1 FOLLOWER]: Refusing update from remote peer 0083994f1c12460eb29337ad7b217e91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:29:51.572928 24015 raft_consensus.cc:1261] T c874cfc77fe64dfe8c8321aa3a09138e P 066cd56b67e94fc4932f012941b09759 [term 1 FOLLOWER]: Refusing update from remote peer 0083994f1c12460eb29337ad7b217e91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:29:51.573405 24374 consensus_queue.cc:1035] T c874cfc77fe64dfe8c8321aa3a09138e P 0083994f1c12460eb29337ad7b217e91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f704d805fef340d68c74e444508c238d" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 36981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:29:51.573606 24384 consensus_queue.cc:1035] T c874cfc77fe64dfe8c8321aa3a09138e P 0083994f1c12460eb29337ad7b217e91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "066cd56b67e94fc4932f012941b09759" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 37391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:29:51.575150 24372 tablet_replica.cc:439] T c874cfc77fe64dfe8c8321aa3a09138e P f704d805fef340d68c74e444508c238d: TxnStatusTablet state changed. Reason: New leader 0083994f1c12460eb29337ad7b217e91. Latest consensus state: current_term: 1 leader_uuid: "0083994f1c12460eb29337ad7b217e91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "066cd56b67e94fc4932f012941b09759" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 37391 } } peers { permanent_uuid: "f704d805fef340d68c74e444508c238d" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 36981 } } peers { permanent_uuid: "0083994f1c12460eb29337ad7b217e91" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 45383 } } }
I20260926 07:29:51.575268 24372 tablet_replica.cc:442] T c874cfc77fe64dfe8c8321aa3a09138e P f704d805fef340d68c74e444508c238d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:29:51.575961 24370 tablet_replica.cc:439] T c874cfc77fe64dfe8c8321aa3a09138e P 066cd56b67e94fc4932f012941b09759: TxnStatusTablet state changed. Reason: New leader 0083994f1c12460eb29337ad7b217e91. Latest consensus state: current_term: 1 leader_uuid: "0083994f1c12460eb29337ad7b217e91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "066cd56b67e94fc4932f012941b09759" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 37391 } } peers { permanent_uuid: "f704d805fef340d68c74e444508c238d" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 36981 } } peers { permanent_uuid: "0083994f1c12460eb29337ad7b217e91" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 45383 } } }
I20260926 07:29:51.576082 24370 tablet_replica.cc:442] T c874cfc77fe64dfe8c8321aa3a09138e P 066cd56b67e94fc4932f012941b09759: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:29:51.578020 24374 tablet_replica.cc:439] T c874cfc77fe64dfe8c8321aa3a09138e P 0083994f1c12460eb29337ad7b217e91: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0083994f1c12460eb29337ad7b217e91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "066cd56b67e94fc4932f012941b09759" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 37391 } } peers { permanent_uuid: "f704d805fef340d68c74e444508c238d" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 36981 } } peers { permanent_uuid: "0083994f1c12460eb29337ad7b217e91" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 45383 } } }
I20260926 07:29:51.578147 24374 tablet_replica.cc:442] T c874cfc77fe64dfe8c8321aa3a09138e P 0083994f1c12460eb29337ad7b217e91: This TxnStatusTablet replica's current role is: LEADER
I20260926 07:29:51.578611 24374 tablet_replica.cc:439] T c874cfc77fe64dfe8c8321aa3a09138e P 0083994f1c12460eb29337ad7b217e91: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0083994f1c12460eb29337ad7b217e91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "066cd56b67e94fc4932f012941b09759" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 37391 } } peers { permanent_uuid: "f704d805fef340d68c74e444508c238d" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 36981 } } peers { permanent_uuid: "0083994f1c12460eb29337ad7b217e91" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 45383 } } }
I20260926 07:29:51.578713 24374 tablet_replica.cc:442] T c874cfc77fe64dfe8c8321aa3a09138e P 0083994f1c12460eb29337ad7b217e91: This TxnStatusTablet replica's current role is: LEADER
I20260926 07:29:51.578913 24372 tablet_replica.cc:439] T c874cfc77fe64dfe8c8321aa3a09138e P f704d805fef340d68c74e444508c238d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0083994f1c12460eb29337ad7b217e91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "066cd56b67e94fc4932f012941b09759" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 37391 } } peers { permanent_uuid: "f704d805fef340d68c74e444508c238d" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 36981 } } peers { permanent_uuid: "0083994f1c12460eb29337ad7b217e91" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 45383 } } }
I20260926 07:29:51.579033 24372 tablet_replica.cc:442] T c874cfc77fe64dfe8c8321aa3a09138e P f704d805fef340d68c74e444508c238d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:29:51.580046 24370 tablet_replica.cc:439] T c874cfc77fe64dfe8c8321aa3a09138e P 066cd56b67e94fc4932f012941b09759: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0083994f1c12460eb29337ad7b217e91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "066cd56b67e94fc4932f012941b09759" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 37391 } } peers { permanent_uuid: "f704d805fef340d68c74e444508c238d" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 36981 } } peers { permanent_uuid: "0083994f1c12460eb29337ad7b217e91" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 45383 } } }
I20260926 07:29:51.580161 24370 tablet_replica.cc:442] T c874cfc77fe64dfe8c8321aa3a09138e P 066cd56b67e94fc4932f012941b09759: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:29:51.581105 24410 mvcc.cc:205] Tried to move back new op lower bound from 7333510314277404672 to 7333510313934340096. Current Snapshot: MvccSnapshot[applied={T|T < 7333510314277404672}]
I20260926 07:29:51.581959 24409 mvcc.cc:205] Tried to move back new op lower bound from 7333510314277404672 to 7333510313934340096. Current Snapshot: MvccSnapshot[applied={T|T < 7333510314277404672 or (T in {7333510314277404672})}]
I20260926 07:29:51.598239 24408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:51.594073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.195:40671 (local address 127.22.53.195:45383)
0926 07:29:51.594224 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:51.594228 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:51.594522 (+ 294us) server_negotiation.cc:407] Connection header received
0926 07:29:51.594738 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:51.594741 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:51.594788 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:51.594872 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:29:51.595218 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:51.595797 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:51.597032 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:51.597272 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:51.597357 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:51.597729 (+ 372us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:29:51.597808 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:29:51.597872 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:51.598010 (+ 138us) server_negotiation.cc:299] Negotiation successful
0926 07:29:51.598062 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 07:29:51.598290 24419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:51.593978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.195:45383 (local address 127.22.53.195:40671)
0926 07:29:51.594418 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:51.594435 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 07:29:51.594569 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:29:51.594911 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:29:51.594915 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:29:51.594924 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 07:29:51.595040 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:51.595047 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:51.595920 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:51.595924 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:51.596892 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:51.596904 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:51.597408 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:51.597412 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:51.597501 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:51.597572 (+ 71us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 07:29:51.597903 (+ 331us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 07:29:51.597911 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 07:29:51.597993 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 07:29:51.598087 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":143,"threads_started":1}
I20260926 07:29:51.607115 24145 raft_consensus.cc:1261] T 52e070984444457bb145ad280db5926d P f704d805fef340d68c74e444508c238d [term 1 FOLLOWER]: Refusing update from remote peer 0083994f1c12460eb29337ad7b217e91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:29:51.607137 24015 raft_consensus.cc:1261] T 52e070984444457bb145ad280db5926d P 066cd56b67e94fc4932f012941b09759 [term 1 FOLLOWER]: Refusing update from remote peer 0083994f1c12460eb29337ad7b217e91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:29:51.607614 24374 consensus_queue.cc:1035] T 52e070984444457bb145ad280db5926d P 0083994f1c12460eb29337ad7b217e91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f704d805fef340d68c74e444508c238d" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 36981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:29:51.607798 24384 consensus_queue.cc:1035] T 52e070984444457bb145ad280db5926d P 0083994f1c12460eb29337ad7b217e91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "066cd56b67e94fc4932f012941b09759" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 37391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:29:51.655064 23856 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58960:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:29:51.655267 23856 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58960:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:29:51.657642 23856 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7a0dca45b0b949f39066155a4f9f160f: Sending DeleteTablet for 3 replicas of tablet 52e070984444457bb145ad280db5926d
I20260926 07:29:51.658422 24135 tablet_service.cc:1514] Processing DeleteTablet for tablet 52e070984444457bb145ad280db5926d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:29:51 UTC) from {username='kudu', principal='kudu/127.22.53.254@KRBTEST.COM'} at 127.0.0.1:45356
I20260926 07:29:51.658422 23995 tablet_service.cc:1514] Processing DeleteTablet for tablet 52e070984444457bb145ad280db5926d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:29:51 UTC) from {username='kudu', principal='kudu/127.22.53.254@KRBTEST.COM'} at 127.0.0.1:47934
I20260926 07:29:51.658505 24274 tablet_service.cc:1514] Processing DeleteTablet for tablet 52e070984444457bb145ad280db5926d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:29:51 UTC) from {username='kudu', principal='kudu/127.22.53.254@KRBTEST.COM'} at 127.0.0.1:57132
I20260926 07:29:51.658929 24426 tablet_replica.cc:330] T 52e070984444457bb145ad280db5926d P 0083994f1c12460eb29337ad7b217e91: stopping tablet replica
I20260926 07:29:51.658998 24425 tablet_replica.cc:330] T 52e070984444457bb145ad280db5926d P f704d805fef340d68c74e444508c238d: stopping tablet replica
I20260926 07:29:51.659092 24426 raft_consensus.cc:2229] T 52e070984444457bb145ad280db5926d P 0083994f1c12460eb29337ad7b217e91 [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:29:51.659106 24425 raft_consensus.cc:2229] T 52e070984444457bb145ad280db5926d P f704d805fef340d68c74e444508c238d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:29:51.659225 24425 raft_consensus.cc:2258] T 52e070984444457bb145ad280db5926d P f704d805fef340d68c74e444508c238d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:29:51.659516 24424 tablet_replica.cc:330] T 52e070984444457bb145ad280db5926d P 066cd56b67e94fc4932f012941b09759: stopping tablet replica
I20260926 07:29:51.659574 24426 raft_consensus.cc:2258] T 52e070984444457bb145ad280db5926d P 0083994f1c12460eb29337ad7b217e91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:29:51.659680 24424 raft_consensus.cc:2229] T 52e070984444457bb145ad280db5926d P 066cd56b67e94fc4932f012941b09759 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:29:51.659821 24424 raft_consensus.cc:2258] T 52e070984444457bb145ad280db5926d P 066cd56b67e94fc4932f012941b09759 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:29:51.660395 24426 ts_tablet_manager.cc:1905] T 52e070984444457bb145ad280db5926d P 0083994f1c12460eb29337ad7b217e91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:29:51.660395 24425 ts_tablet_manager.cc:1905] T 52e070984444457bb145ad280db5926d P f704d805fef340d68c74e444508c238d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:29:51.660583 24424 ts_tablet_manager.cc:1905] T 52e070984444457bb145ad280db5926d P 066cd56b67e94fc4932f012941b09759: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:29:51.663738 24426 ts_tablet_manager.cc:1918] T 52e070984444457bb145ad280db5926d P 0083994f1c12460eb29337ad7b217e91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 07:29:51.663724 24424 ts_tablet_manager.cc:1918] T 52e070984444457bb145ad280db5926d P 066cd56b67e94fc4932f012941b09759: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 07:29:51.663838 24424 log.cc:1196] T 52e070984444457bb145ad280db5926d P 066cd56b67e94fc4932f012941b09759: Deleting WAL directory at /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407783529717-22743-0/minicluster-data/ts-0/wal/wals/52e070984444457bb145ad280db5926d
I20260926 07:29:51.663838 24426 log.cc:1196] T 52e070984444457bb145ad280db5926d P 0083994f1c12460eb29337ad7b217e91: Deleting WAL directory at /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407783529717-22743-0/minicluster-data/ts-2/wal/wals/52e070984444457bb145ad280db5926d
I20260926 07:29:51.664115 24424 ts_tablet_manager.cc:1939] T 52e070984444457bb145ad280db5926d P 066cd56b67e94fc4932f012941b09759: Deleting consensus metadata
I20260926 07:29:51.664115 24426 ts_tablet_manager.cc:1939] T 52e070984444457bb145ad280db5926d P 0083994f1c12460eb29337ad7b217e91: Deleting consensus metadata
I20260926 07:29:51.664636 24425 ts_tablet_manager.cc:1918] T 52e070984444457bb145ad280db5926d P f704d805fef340d68c74e444508c238d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 07:29:51.664835 24425 log.cc:1196] T 52e070984444457bb145ad280db5926d P f704d805fef340d68c74e444508c238d: Deleting WAL directory at /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407783529717-22743-0/minicluster-data/ts-1/wal/wals/52e070984444457bb145ad280db5926d
I20260926 07:29:51.665172 23845 catalog_manager.cc:4841] TS 0083994f1c12460eb29337ad7b217e91 (127.22.53.195:45383): tablet 52e070984444457bb145ad280db5926d (table test-table [id=5b8816b8199544de852b87c2426d4efa]) successfully deleted
I20260926 07:29:51.665524 23845 catalog_manager.cc:4841] TS 066cd56b67e94fc4932f012941b09759 (127.22.53.193:37391): tablet 52e070984444457bb145ad280db5926d (table test-table [id=5b8816b8199544de852b87c2426d4efa]) successfully deleted
I20260926 07:29:51.665860 24425 ts_tablet_manager.cc:1939] T 52e070984444457bb145ad280db5926d P f704d805fef340d68c74e444508c238d: Deleting consensus metadata
I20260926 07:29:51.666666 23844 catalog_manager.cc:4841] TS f704d805fef340d68c74e444508c238d (127.22.53.194:36981): tablet 52e070984444457bb145ad280db5926d (table test-table [id=5b8816b8199544de852b87c2426d4efa]) successfully deleted
Sep 26 07:29:51 dist-test-slave-wdjc krb5kdc[23813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407790, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.22.53.193@KRBTEST.COM
I20260926 07:29:51.672262 24428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:51.660034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47938 (local address 127.22.53.193:37391)
0926 07:29:51.660803 (+ 769us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:51.660808 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:51.660822 (+ 14us) server_negotiation.cc:407] Connection header received
0926 07:29:51.660863 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:51.660866 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:51.660911 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:51.661934 (+ 1023us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:51.662180 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:51.662975 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:51.664271 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:51.664710 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:51.665290 (+ 580us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:51.667904 (+ 2614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:51.667917 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:51.667934 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:29:51.667963 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:51.670059 (+ 2096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:51.670651 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:51.670656 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:51.670658 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:51.670751 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:51.671087 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:51.671090 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:51.671092 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:51.671417 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:51.671594 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:51.671942 (+ 348us) server_negotiation.cc:299] Negotiation successful
0926 07:29:51.672081 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":695,"thread_start_us":83,"threads_started":1}
W20260926 07:29:51.672884 24055 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47938
I20260926 07:29:51.684319 24400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:51.675004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58982 (local address 127.22.53.254:44201)
0926 07:29:51.675183 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:51.675189 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:51.675299 (+ 110us) server_negotiation.cc:407] Connection header received
0926 07:29:51.675473 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:51.675476 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:51.675532 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:51.675630 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:51.676385 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:51.677057 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:51.677950 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:51.678204 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:51.678352 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:51.679910 (+ 1558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:51.679913 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:51.679916 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:29:51.679947 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:51.682111 (+ 2164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:51.682775 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:51.682784 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:51.682786 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:51.682884 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:51.683225 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:51.683229 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:51.683231 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:51.683457 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:51.683687 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:51.683968 (+ 281us) server_negotiation.cc:299] Negotiation successful
0926 07:29:51.684124 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260926 07:29:51.684939 23856 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58982
I20260926 07:29:51.686342 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 23930
I20260926 07:29:51.692837 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 24070
I20260926 07:29:51.699548 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 24210
I20260926 07:29:51.707180 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 23826
2026-09-26T07:29:51Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3393 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407783529717-22743-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:29:51 dist-test-slave-wdjc krb5kdc[24434](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:29:51 dist-test-slave-wdjc krb5kdc[24434](info): set up 2 sockets
Sep 26 07:29:51 dist-test-slave-wdjc krb5kdc[24434](info): commencing operation
krb5kdc: starting...
W20260926 07:29:52.756997 22743 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407783529717-22743-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407783529717-22743-0/krb5kdc/test-user.keytab.
Sep 26 07:29:52 dist-test-slave-wdjc krb5kdc[24434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407792, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:29:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:29:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.53.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.53.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.53.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab.
Entry for principal kudu/127.22.53.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab.
I20260926 07:29:52.888299 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407783529717-22743-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407783529717-22743-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407783529717-22743-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.53.254:38529
--webserver_interface=127.22.53.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab
--principal=kudu/127.22.53.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:42621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.53.254:38529
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:29:53.006181 24447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:29:53.006541 24447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:29:53.006620 24447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:29:53.010921 24447 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:29:53.011063 24447 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 07:29:53.011094 24447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:29:53.011129 24447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:29:53.011193 24447 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:29:53.011257 24447 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:29:53.017155 24447 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:42621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407783529717-22743-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407783529717-22743-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.53.254:38529
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab
--principal=kudu/127.22.53.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.53.254:38529
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.53.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24447
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407783529717-22743-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:29:53.018404 24447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:29:53.019531 24447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:29:53.029098 24453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:29:53.029245 24447 server_base.cc:1034] running on GCE node
W20260926 07:29:53.029098 24455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:29:53.029462 24452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:29:53.030181 24447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:29:53.031425 24447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:29:53.032666 24447 hybrid_clock.cc:648] HybridClock initialized: now 1790407793032648 us; error 30 us; skew 500 ppm
Sep 26 07:29:53 dist-test-slave-wdjc krb5kdc[24434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407793, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:29:53.035890 24447 init.cc:377] Logged in from keytab as kudu/127.22.53.254@KRBTEST.COM (short username kudu)
I20260926 07:29:53.037029 24447 webserver.cc:460] Webserver started at http://127.22.53.254:34877/ using document root <none> and password file <none>
I20260926 07:29:53.037611 24447 fs_manager.cc:359] Metadata directory not provided
I20260926 07:29:53.037679 24447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:29:53.037865 24447 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:29:53.040179 24447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407783529717-22743-0/minicluster-data/master-0/data/instance:
uuid: "96456e5ef902415d9f99e829c7a793a1"
format_stamp: "Formatted at 2026-09-26 07:29:53 on dist-test-slave-wdjc"
server_key: "f1fd1c30d12ce815e63b2c3902fa0c57"
server_key_iv: "62854318d46d6ac865854b73e015893f"
server_key_version: "encryptionkey@0"
I20260926 07:29:53.040889 24447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407783529717-22743-0/minicluster-data/master-0/wal/instance:
uuid: "96456e5ef902415d9f99e829c7a793a1"
format_stamp: "Formatted at 2026-09-26 07:29:53 on dist-test-slave-wdjc"
server_key: "f1fd1c30d12ce815e63b2c3902fa0c57"
server_key_iv: "62854318d46d6ac865854b73e015893f"
server_key_version: "encryptionkey@0"
I20260926 07:29:53.045277 24447 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 07:29:53.047997 24462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:53.049522 24447 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:29:53.049767 24447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407783529717-22743-0/minicluster-data/master-0/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407783529717-22743-0/minicluster-data/master-0/wal
uuid: "96456e5ef902415d9f99e829c7a793a1"
format_stamp: "Formatted at 2026-09-26 07:29:53 on dist-test-slave-wdjc"
server_key: "f1fd1c30d12ce815e63b2c3902fa0c57"
server_key_iv: "62854318d46d6ac865854b73e015893f"
server_key_version: "encryptionkey@0"
I20260926 07:29:53.049918 24447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407783529717-22743-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407783529717-22743-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407783529717-22743-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:29:53.078189 24447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:29:53.081929 24447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:29:53.082190 24447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:29:53.091691 24447 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.254:38529
I20260926 07:29:53.091701 24524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.254:38529 every 8 connection(s)
I20260926 07:29:53.092861 24447 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
I20260926 07:29:53.095969 24525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:29:53.096313 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 24447
I20260926 07:29:53.096429 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407783529717-22743-0/minicluster-data/master-0/wal/instance
I20260926 07:29:53.096750 22743 external_mini_cluster.cc:1428] Setting key dbd7361afb06c23fcc11061328d0267d
I20260926 07:29:53.103227 24525 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96456e5ef902415d9f99e829c7a793a1: Bootstrap starting.
I20260926 07:29:53.105851 24525 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 96456e5ef902415d9f99e829c7a793a1: Neither blocks nor log segments found. Creating new log.
Sep 26 07:29:53 dist-test-slave-wdjc krb5kdc[24434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407792, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.53.254@KRBTEST.COM
I20260926 07:29:53.106711 24525 log.cc:824] T 00000000000000000000000000000000 P 96456e5ef902415d9f99e829c7a793a1: Log is configured to *not* fsync() on all Append() calls
I20260926 07:29:53.108877 24525 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96456e5ef902415d9f99e829c7a793a1: No bootstrap required, opened a new log
I20260926 07:29:53.112085 24525 raft_consensus.cc:348] T 00000000000000000000000000000000 P 96456e5ef902415d9f99e829c7a793a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96456e5ef902415d9f99e829c7a793a1" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 38529 } }
I20260926 07:29:53.112287 24528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:53.098378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33190 (local address 127.22.53.254:38529)
0926 07:29:53.098882 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:53.098897 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:53.098936 (+ 39us) server_negotiation.cc:407] Connection header received
0926 07:29:53.099812 (+ 876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:53.099835 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:53.100208 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:53.100592 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:53.101451 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:53.102468 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:53.103364 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:53.103881 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:53.104060 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:53.106787 (+ 2727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:53.106792 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:53.106809 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:29:53.106864 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:53.109289 (+ 2425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:53.109795 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:53.109800 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:53.109805 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:53.109923 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:53.110240 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:53.110244 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:53.110246 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:53.110667 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:53.110883 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:53.111187 (+ 304us) server_negotiation.cc:299] Negotiation successful
0926 07:29:53.111504 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":139,"threads_started":1}
I20260926 07:29:53.112370 24525 raft_consensus.cc:374] T 00000000000000000000000000000000 P 96456e5ef902415d9f99e829c7a793a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:29:53.112435 24525 raft_consensus.cc:729] T 00000000000000000000000000000000 P 96456e5ef902415d9f99e829c7a793a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96456e5ef902415d9f99e829c7a793a1, State: Initialized, Role: FOLLOWER
I20260926 07:29:53.112977 24525 consensus_queue.cc:260] T 00000000000000000000000000000000 P 96456e5ef902415d9f99e829c7a793a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96456e5ef902415d9f99e829c7a793a1" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 38529 } }
I20260926 07:29:53.113137 24525 raft_consensus.cc:388] T 00000000000000000000000000000000 P 96456e5ef902415d9f99e829c7a793a1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:29:53.113190 24525 raft_consensus.cc:482] T 00000000000000000000000000000000 P 96456e5ef902415d9f99e829c7a793a1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:29:53.113273 24525 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 96456e5ef902415d9f99e829c7a793a1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:29:53.114271 24525 raft_consensus.cc:504] T 00000000000000000000000000000000 P 96456e5ef902415d9f99e829c7a793a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96456e5ef902415d9f99e829c7a793a1" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 38529 } }
I20260926 07:29:53.114650 24525 leader_election.cc:302] T 00000000000000000000000000000000 P 96456e5ef902415d9f99e829c7a793a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 96456e5ef902415d9f99e829c7a793a1; no voters:
I20260926 07:29:53.115036 24525 leader_election.cc:288] T 00000000000000000000000000000000 P 96456e5ef902415d9f99e829c7a793a1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:29:53.115149 24530 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 96456e5ef902415d9f99e829c7a793a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:29:53.115408 24530 raft_consensus.cc:686] T 00000000000000000000000000000000 P 96456e5ef902415d9f99e829c7a793a1 [term 1 LEADER]: Becoming Leader. State: Replica: 96456e5ef902415d9f99e829c7a793a1, State: Running, Role: LEADER
I20260926 07:29:53.115782 24530 consensus_queue.cc:237] T 00000000000000000000000000000000 P 96456e5ef902415d9f99e829c7a793a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96456e5ef902415d9f99e829c7a793a1" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 38529 } }
I20260926 07:29:53.116305 24525 sys_catalog.cc:564] T 00000000000000000000000000000000 P 96456e5ef902415d9f99e829c7a793a1 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:29:53.117369 24531 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96456e5ef902415d9f99e829c7a793a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "96456e5ef902415d9f99e829c7a793a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96456e5ef902415d9f99e829c7a793a1" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 38529 } } }
I20260926 07:29:53.117496 24531 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96456e5ef902415d9f99e829c7a793a1 [sys.catalog]: This master's current role is: LEADER
I20260926 07:29:53.117859 24532 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96456e5ef902415d9f99e829c7a793a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 96456e5ef902415d9f99e829c7a793a1. Latest consensus state: current_term: 1 leader_uuid: "96456e5ef902415d9f99e829c7a793a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96456e5ef902415d9f99e829c7a793a1" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 38529 } } }
I20260926 07:29:53.118026 24532 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96456e5ef902415d9f99e829c7a793a1 [sys.catalog]: This master's current role is: LEADER
I20260926 07:29:53.118367 24539 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:29:53.121742 24539 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:29:53.127213 24539 catalog_manager.cc:1347] Generated new cluster ID: 31d6f9fd4fcc4999a22a4d67175db33b
I20260926 07:29:53.127311 24539 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:29:53.141228 24539 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:29:53.142647 24539 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:29:53.153563 24539 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 96456e5ef902415d9f99e829c7a793a1: Generated new TSK 0
I20260926 07:29:53.154529 24539 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.53.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.53.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.53.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.193.keytab.
Entry for principal kudu/127.22.53.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.193.keytab.
I20260926 07:29:53.195887 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407783529717-22743-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407783529717-22743-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407783529717-22743-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407783529717-22743-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.53.193:0
--local_ip_for_outbound_sockets=127.22.53.193
--webserver_interface=127.22.53.193
--webserver_port=0
--tserver_master_addrs=127.22.53.254:38529
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.193.keytab
--principal=kudu/127.22.53.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:42621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:29:53.317140 24551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:29:53.317562 24551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:29:53.317632 24551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:29:53.322013 24551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:29:53.322119 24551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:29:53.322199 24551 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 07:29:53.322243 24551 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 07:29:53.322314 24551 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.53.193
I20260926 07:29:53.327955 24551 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:42621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407783529717-22743-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407783529717-22743-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.193.keytab
--principal=kudu/127.22.53.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.53.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407783529717-22743-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.53.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.53.254:38529
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24551
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.53.193
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407783529717-22743-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:29:53.329263 24551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:29:53.330399 24551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:29:53.341353 24559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:29:53.341372 24557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:29:53.341347 24556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:29:53.341665 24551 server_base.cc:1034] running on GCE node
I20260926 07:29:53.342206 24551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:29:53.342938 24551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:29:53.344116 24551 hybrid_clock.cc:648] HybridClock initialized: now 1790407793344078 us; error 57 us; skew 500 ppm
Sep 26 07:29:53 dist-test-slave-wdjc krb5kdc[24434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407793, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:29:53.348187 24551 init.cc:377] Logged in from keytab as kudu/127.22.53.193@KRBTEST.COM (short username kudu)
I20260926 07:29:53.349517 24551 webserver.cc:460] Webserver started at http://127.22.53.193:44603/ using document root <none> and password file <none>
I20260926 07:29:53.350199 24551 fs_manager.cc:359] Metadata directory not provided
I20260926 07:29:53.350323 24551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:29:53.350597 24551 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:29:53.353184 24551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407783529717-22743-0/minicluster-data/ts-0/data/instance:
uuid: "3a69b617c9f843b2a9001f6d598c63e8"
format_stamp: "Formatted at 2026-09-26 07:29:53 on dist-test-slave-wdjc"
server_key: "0c561c67e9246b4b611cadc3e0d4c67c"
server_key_iv: "fd71460da84fc03d47d4787c887ff01f"
server_key_version: "encryptionkey@0"
I20260926 07:29:53.353868 24551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407783529717-22743-0/minicluster-data/ts-0/wal/instance:
uuid: "3a69b617c9f843b2a9001f6d598c63e8"
format_stamp: "Formatted at 2026-09-26 07:29:53 on dist-test-slave-wdjc"
server_key: "0c561c67e9246b4b611cadc3e0d4c67c"
server_key_iv: "fd71460da84fc03d47d4787c887ff01f"
server_key_version: "encryptionkey@0"
I20260926 07:29:53.358911 24551 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 07:29:53.361943 24566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:53.363517 24551 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 07:29:53.363878 24551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407783529717-22743-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407783529717-22743-0/minicluster-data/ts-0/wal
uuid: "3a69b617c9f843b2a9001f6d598c63e8"
format_stamp: "Formatted at 2026-09-26 07:29:53 on dist-test-slave-wdjc"
server_key: "0c561c67e9246b4b611cadc3e0d4c67c"
server_key_iv: "fd71460da84fc03d47d4787c887ff01f"
server_key_version: "encryptionkey@0"
I20260926 07:29:53.364070 24551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407783529717-22743-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407783529717-22743-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407783529717-22743-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:29:53.383200 24551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:29:53.386927 24551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:29:53.387156 24551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:29:53.388777 24551 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:29:53.388855 24551 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:53.388931 24551 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:29:53.388974 24551 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 07:29:53 dist-test-slave-wdjc krb5kdc[24434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407793, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.193@KRBTEST.COM for kudu/127.22.53.254@KRBTEST.COM
I20260926 07:29:53.400425 24551 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.193:38223
I20260926 07:29:53.400660 24681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.193:38223 every 8 connection(s)
I20260926 07:29:53.401463 24551 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407783529717-22743-0/minicluster-data/ts-0/data/info.pb
I20260926 07:29:53.403586 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 24551
I20260926 07:29:53.403738 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407783529717-22743-0/minicluster-data/ts-0/wal/instance
I20260926 07:29:53.404022 22743 external_mini_cluster.cc:1428] Setting key 267c364dc30e41614b3687e9cafeec56
I20260926 07:29:53.404259 24528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:53.390401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.193:53989 (local address 127.22.53.254:38529)
0926 07:29:53.390575 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:53.390581 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:53.391883 (+ 1302us) server_negotiation.cc:407] Connection header received
0926 07:29:53.393281 (+ 1398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:53.393288 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:53.393365 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:53.393475 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:53.394982 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:53.395527 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:53.396831 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:53.397073 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:53.397191 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:53.400000 (+ 2809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:53.400004 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:53.400007 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:29:53.400045 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:53.401820 (+ 1775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:53.402488 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:53.402493 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:53.402495 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:53.402615 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:53.402944 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:53.402947 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:53.402948 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:53.403144 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:53.403227 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:53.403899 (+ 672us) server_negotiation.cc:299] Negotiation successful
0926 07:29:53.404056 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 07:29:53.405280 24577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:53.390944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:38529 (local address 127.22.53.193:53989)
0926 07:29:53.391530 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:53.391584 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 07:29:53.392935 (+ 1351us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:29:53.393690 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:29:53.393702 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:29:53.394091 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:29:53.394812 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:53.394831 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:53.395668 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:53.395671 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:53.396689 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:53.396700 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:53.397240 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:53.397244 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:53.397436 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:53.398160 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:29:53.398185 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:29:53.399797 (+ 1612us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:29:53.401953 (+ 2156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:29:53.401961 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:29:53.401978 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:29:53.402366 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:29:53.402717 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:29:53.402719 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:29:53.402720 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:29:53.402860 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:29:53.403267 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:29:53.403280 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:29:53.403565 (+ 285us) client_negotiation.cc:769] Sending connection context
0926 07:29:53.403845 (+ 280us) client_negotiation.cc:240] Negotiation successful
0926 07:29:53.404238 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":227,"threads_started":1}
I20260926 07:29:53.406752 24682 heartbeater.cc:347] Connected to a master server at 127.22.53.254:38529
I20260926 07:29:53.406947 24682 heartbeater.cc:464] Registering TS with master...
I20260926 07:29:53.407558 24682 heartbeater.cc:511] Master 127.22.53.254:38529 requested a full tablet report, sending...
I20260926 07:29:53.409487 24479 ts_manager.cc:194] Registered new tserver with Master: 3a69b617c9f843b2a9001f6d598c63e8 (127.22.53.193:38223)
I20260926 07:29:53.411240 24479 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.53.193@KRBTEST.COM'} at 127.22.53.193:53989
WARNING: no policy specified for kudu/127.22.53.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.53.194@KRBTEST.COM" created.
I20260926 07:29:53.421861 24528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:53.412585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.193:52911 (local address 127.22.53.254:38529)
0926 07:29:53.412789 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:53.412795 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:53.413095 (+ 300us) server_negotiation.cc:407] Connection header received
0926 07:29:53.413232 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:53.413235 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:53.413290 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:53.413390 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:53.414059 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:53.414745 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:53.415962 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:53.416193 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:53.416335 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:53.417744 (+ 1409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:53.417748 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:53.417751 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:29:53.418093 (+ 342us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:53.419729 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:53.420353 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:53.420359 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:53.420362 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:53.420476 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:53.420825 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:53.420830 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:53.420833 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:53.421110 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:53.421226 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:53.421532 (+ 306us) server_negotiation.cc:299] Negotiation successful
0926 07:29:53.421678 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260926 07:29:53.421917 24689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:53.412475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:38529 (local address 127.22.53.193:52911)
0926 07:29:53.412979 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:53.412996 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 07:29:53.413120 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:29:53.413437 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:29:53.413440 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:29:53.413694 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:29:53.413855 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:53.413879 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:53.414900 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:53.414908 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:53.415833 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:53.415843 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:53.416373 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:53.416380 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:53.416517 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:53.417145 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:29:53.417158 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:29:53.417601 (+ 443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:29:53.419869 (+ 2268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:29:53.419875 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:29:53.419881 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:29:53.420193 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:29:53.420631 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:29:53.420633 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:29:53.420635 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:29:53.420724 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:29:53.421246 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:29:53.421249 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:29:53.421347 (+ 98us) client_negotiation.cc:769] Sending connection context
0926 07:29:53.421449 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 07:29:53.421604 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":159,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.53.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.194.keytab.
Entry for principal kudu/127.22.53.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.194.keytab.
I20260926 07:29:53.436084 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407783529717-22743-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407783529717-22743-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407783529717-22743-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407783529717-22743-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.53.194:0
--local_ip_for_outbound_sockets=127.22.53.194
--webserver_interface=127.22.53.194
--webserver_port=0
--tserver_master_addrs=127.22.53.254:38529
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.194.keytab
--principal=kudu/127.22.53.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:42621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:29:53.555342 24691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:29:53.555649 24691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:29:53.555701 24691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:29:53.559996 24691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:29:53.560086 24691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:29:53.560132 24691 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 07:29:53.560146 24691 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 07:29:53.560180 24691 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.53.194
I20260926 07:29:53.565966 24691 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:42621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407783529717-22743-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407783529717-22743-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.194.keytab
--principal=kudu/127.22.53.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.53.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407783529717-22743-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.53.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.53.254:38529
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24691
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.53.194
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407783529717-22743-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:29:53.567130 24691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:29:53.568279 24691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:29:53.579126 24697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:29:53.579195 24699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:29:53.579336 24696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:29:53.579495 24691 server_base.cc:1034] running on GCE node
I20260926 07:29:53.579988 24691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:29:53.580888 24691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:29:53.582093 24691 hybrid_clock.cc:648] HybridClock initialized: now 1790407793582076 us; error 37 us; skew 500 ppm
Sep 26 07:29:53 dist-test-slave-wdjc krb5kdc[24434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407793, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:29:53.585106 24691 init.cc:377] Logged in from keytab as kudu/127.22.53.194@KRBTEST.COM (short username kudu)
I20260926 07:29:53.586043 24691 webserver.cc:460] Webserver started at http://127.22.53.194:46683/ using document root <none> and password file <none>
I20260926 07:29:53.586597 24691 fs_manager.cc:359] Metadata directory not provided
I20260926 07:29:53.586659 24691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:29:53.586841 24691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:29:53.589072 24691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407783529717-22743-0/minicluster-data/ts-1/data/instance:
uuid: "ffbd74368f6e483abebb4ff75b582ba0"
format_stamp: "Formatted at 2026-09-26 07:29:53 on dist-test-slave-wdjc"
server_key: "24281da35d4ef46b8a29d7fafd366568"
server_key_iv: "0c9f2d069bca3eecb37ef1792301b49d"
server_key_version: "encryptionkey@0"
I20260926 07:29:53.589618 24691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407783529717-22743-0/minicluster-data/ts-1/wal/instance:
uuid: "ffbd74368f6e483abebb4ff75b582ba0"
format_stamp: "Formatted at 2026-09-26 07:29:53 on dist-test-slave-wdjc"
server_key: "24281da35d4ef46b8a29d7fafd366568"
server_key_iv: "0c9f2d069bca3eecb37ef1792301b49d"
server_key_version: "encryptionkey@0"
I20260926 07:29:53.593776 24691 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 07:29:53.596511 24706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:53.598040 24691 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:29:53.598157 24691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407783529717-22743-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407783529717-22743-0/minicluster-data/ts-1/wal
uuid: "ffbd74368f6e483abebb4ff75b582ba0"
format_stamp: "Formatted at 2026-09-26 07:29:53 on dist-test-slave-wdjc"
server_key: "24281da35d4ef46b8a29d7fafd366568"
server_key_iv: "0c9f2d069bca3eecb37ef1792301b49d"
server_key_version: "encryptionkey@0"
I20260926 07:29:53.598282 24691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407783529717-22743-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407783529717-22743-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407783529717-22743-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:29:53.614777 24691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:29:53.619311 24691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:29:53.619560 24691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:29:53.621829 24691 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:29:53.621915 24691 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:53.621984 24691 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:29:53.622007 24691 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 07:29:53 dist-test-slave-wdjc krb5kdc[24434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407793, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.194@KRBTEST.COM for kudu/127.22.53.254@KRBTEST.COM
I20260926 07:29:53.635044 24691 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.194:41601
I20260926 07:29:53.635105 24821 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.194:41601 every 8 connection(s)
I20260926 07:29:53.636461 24691 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407783529717-22743-0/minicluster-data/ts-1/data/info.pb
I20260926 07:29:53.639907 24528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:53.624521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.194:42569 (local address 127.22.53.254:38529)
0926 07:29:53.624709 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:53.624716 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:53.624739 (+ 23us) server_negotiation.cc:407] Connection header received
0926 07:29:53.625574 (+ 835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:53.625579 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:53.625645 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:53.625703 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:53.628338 (+ 2635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:53.629441 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:53.631478 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:53.631891 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:53.632055 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:53.635376 (+ 3321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:53.635381 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:53.635384 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:29:53.635421 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:53.637339 (+ 1918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:53.638089 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:53.638093 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:53.638096 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:53.638222 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:53.638640 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:53.638645 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:53.638647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:53.638875 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:53.639007 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:53.639572 (+ 565us) server_negotiation.cc:299] Negotiation successful
0926 07:29:53.639723 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 07:29:53.640944 24715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:53.623333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:38529 (local address 127.22.53.194:42569)
0926 07:29:53.623947 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:53.624002 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 07:29:53.625273 (+ 1271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:29:53.626194 (+ 921us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:29:53.626204 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:29:53.627312 (+ 1108us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:29:53.628075 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:53.628098 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:53.629620 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:53.629625 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:53.631309 (+ 1684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:53.631322 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:53.632089 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:53.632092 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:53.632253 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:53.633168 (+ 915us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:29:53.633196 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:29:53.635151 (+ 1955us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:29:53.637529 (+ 2378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:29:53.637535 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:29:53.637553 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:29:53.637958 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:29:53.638352 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:29:53.638355 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:29:53.638357 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:29:53.638530 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:29:53.639004 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:29:53.639012 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:29:53.639303 (+ 291us) client_negotiation.cc:769] Sending connection context
0926 07:29:53.639524 (+ 221us) client_negotiation.cc:240] Negotiation successful
0926 07:29:53.639917 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":165,"threads_started":1}
I20260926 07:29:53.642403 24822 heartbeater.cc:347] Connected to a master server at 127.22.53.254:38529
I20260926 07:29:53.642567 24822 heartbeater.cc:464] Registering TS with master...
I20260926 07:29:53.643231 24822 heartbeater.cc:511] Master 127.22.53.254:38529 requested a full tablet report, sending...
I20260926 07:29:53.643690 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 24691
I20260926 07:29:53.643826 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407783529717-22743-0/minicluster-data/ts-1/wal/instance
I20260926 07:29:53.644207 22743 external_mini_cluster.cc:1428] Setting key 0e0237897764de41a003fdd0d71c4f42
I20260926 07:29:53.644560 24479 ts_manager.cc:194] Registered new tserver with Master: ffbd74368f6e483abebb4ff75b582ba0 (127.22.53.194:41601)
I20260926 07:29:53.645354 24479 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.53.194@KRBTEST.COM'} at 127.22.53.194:42569
I20260926 07:29:53.654150 24528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:53.645084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.194:51913 (local address 127.22.53.254:38529)
0926 07:29:53.645233 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:53.645237 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:53.645938 (+ 701us) server_negotiation.cc:407] Connection header received
0926 07:29:53.646075 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:53.646078 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:53.646127 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:53.646221 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:53.646689 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:53.647206 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:53.648315 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:53.648658 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:53.648756 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:53.650269 (+ 1513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:53.650273 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:53.650275 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:29:53.650299 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:53.652000 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:53.652504 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:53.652508 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:53.652509 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:53.652656 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:53.652991 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:53.652995 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:53.652997 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:53.653380 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:53.653512 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:53.653826 (+ 314us) server_negotiation.cc:299] Negotiation successful
0926 07:29:53.653965 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 07:29:53.654163 24828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:53.645002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:38529 (local address 127.22.53.194:51913)
0926 07:29:53.645836 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:53.645850 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:29:53.645957 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:29:53.646237 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:29:53.646240 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:29:53.646474 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:29:53.646574 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:53.646580 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:53.647342 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:53.647345 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:53.648181 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:53.648193 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:53.648795 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:53.648800 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:53.648917 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:53.649530 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:29:53.649550 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:29:53.650085 (+ 535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:29:53.652139 (+ 2054us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:29:53.652142 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:29:53.652145 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:29:53.652375 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:29:53.652788 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:29:53.652790 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:29:53.652792 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:29:53.652880 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:29:53.653535 (+ 655us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:29:53.653537 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:29:53.653636 (+ 99us) client_negotiation.cc:769] Sending connection context
0926 07:29:53.653768 (+ 132us) client_negotiation.cc:240] Negotiation successful
0926 07:29:53.653965 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":748,"thread_start_us":136,"threads_started":1}
WARNING: no policy specified for kudu/127.22.53.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.53.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.53.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.195.keytab.
Entry for principal kudu/127.22.53.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.195.keytab.
I20260926 07:29:53.676805 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407783529717-22743-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407783529717-22743-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407783529717-22743-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407783529717-22743-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.53.195:0
--local_ip_for_outbound_sockets=127.22.53.195
--webserver_interface=127.22.53.195
--webserver_port=0
--tserver_master_addrs=127.22.53.254:38529
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.195.keytab
--principal=kudu/127.22.53.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:42621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:29:53.799338 24831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:29:53.799746 24831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:29:53.799849 24831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:29:53.804385 24831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:29:53.804508 24831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:29:53.804630 24831 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 07:29:53.804683 24831 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 07:29:53.804766 24831 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.53.195
I20260926 07:29:53.811231 24831 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:42621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407783529717-22743-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407783529717-22743-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.195.keytab
--principal=kudu/127.22.53.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.53.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407783529717-22743-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.53.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.53.254:38529
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24831
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.53.195
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407783529717-22743-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:29:53.812557 24831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:29:53.813676 24831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:29:53.824532 24837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:29:53.824517 24839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:29:53.824517 24836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:29:53.824829 24831 server_base.cc:1034] running on GCE node
I20260926 07:29:53.825382 24831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:29:53.826234 24831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:29:53.827453 24831 hybrid_clock.cc:648] HybridClock initialized: now 1790407793827408 us; error 67 us; skew 500 ppm
Sep 26 07:29:53 dist-test-slave-wdjc krb5kdc[24434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407793, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:29:53.830646 24831 init.cc:377] Logged in from keytab as kudu/127.22.53.195@KRBTEST.COM (short username kudu)
I20260926 07:29:53.831763 24831 webserver.cc:460] Webserver started at http://127.22.53.195:33323/ using document root <none> and password file <none>
I20260926 07:29:53.832379 24831 fs_manager.cc:359] Metadata directory not provided
I20260926 07:29:53.832476 24831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:29:53.832747 24831 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:29:53.834964 24831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407783529717-22743-0/minicluster-data/ts-2/data/instance:
uuid: "66291c8b70dc46a6acbfdf3c9c475500"
format_stamp: "Formatted at 2026-09-26 07:29:53 on dist-test-slave-wdjc"
server_key: "6be3db24ad3a602d179eacfd1c9f4a8f"
server_key_iv: "8ef5dba610beddf07c21b9f10e5f3409"
server_key_version: "encryptionkey@0"
I20260926 07:29:53.835575 24831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407783529717-22743-0/minicluster-data/ts-2/wal/instance:
uuid: "66291c8b70dc46a6acbfdf3c9c475500"
format_stamp: "Formatted at 2026-09-26 07:29:53 on dist-test-slave-wdjc"
server_key: "6be3db24ad3a602d179eacfd1c9f4a8f"
server_key_iv: "8ef5dba610beddf07c21b9f10e5f3409"
server_key_version: "encryptionkey@0"
I20260926 07:29:53.839725 24831 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 07:29:53.842649 24846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:53.844204 24831 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 07:29:53.844383 24831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407783529717-22743-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407783529717-22743-0/minicluster-data/ts-2/wal
uuid: "66291c8b70dc46a6acbfdf3c9c475500"
format_stamp: "Formatted at 2026-09-26 07:29:53 on dist-test-slave-wdjc"
server_key: "6be3db24ad3a602d179eacfd1c9f4a8f"
server_key_iv: "8ef5dba610beddf07c21b9f10e5f3409"
server_key_version: "encryptionkey@0"
I20260926 07:29:53.844524 24831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407783529717-22743-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407783529717-22743-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407783529717-22743-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:29:53.871277 24831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:29:53.874655 24831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:29:53.874879 24831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:29:53.876449 24831 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:29:53.876526 24831 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:53.876652 24831 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:29:53.876700 24831 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 07:29:53 dist-test-slave-wdjc krb5kdc[24434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407793, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.195@KRBTEST.COM for kudu/127.22.53.254@KRBTEST.COM
I20260926 07:29:53.888561 24831 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.195:38915
I20260926 07:29:53.888720 24961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.195:38915 every 8 connection(s)
I20260926 07:29:53.889667 24831 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407783529717-22743-0/minicluster-data/ts-2/data/info.pb
I20260926 07:29:53.891743 24528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:53.877388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.195:54731 (local address 127.22.53.254:38529)
0926 07:29:53.877569 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:53.877576 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:53.878511 (+ 935us) server_negotiation.cc:407] Connection header received
0926 07:29:53.879836 (+ 1325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:53.879842 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:53.879918 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:53.880023 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:53.881510 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:53.882120 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:53.883412 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:53.883726 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:53.883866 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:53.887457 (+ 3591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:53.887461 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:53.887463 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:29:53.887518 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:53.889381 (+ 1863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:53.890073 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:53.890077 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:53.890080 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:53.890195 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:53.890587 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:53.890591 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:53.890594 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:53.890882 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:53.890981 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:53.891494 (+ 513us) server_negotiation.cc:299] Negotiation successful
0926 07:29:53.891611 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 07:29:53.892648 24855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:53.877834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:38529 (local address 127.22.53.195:54731)
0926 07:29:53.878326 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:53.878381 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 07:29:53.879506 (+ 1125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:29:53.880298 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:29:53.880310 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:29:53.880796 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:29:53.881332 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:53.881347 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:53.882270 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:53.882275 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:53.883260 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:53.883271 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:53.883925 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:53.883929 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:53.884116 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:53.884914 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:29:53.884933 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:29:53.887240 (+ 2307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:29:53.889522 (+ 2282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:29:53.889528 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:29:53.889543 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:29:53.889932 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:29:53.890320 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:29:53.890323 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:29:53.890325 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:29:53.890486 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:29:53.890988 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:29:53.890993 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:29:53.891229 (+ 236us) client_negotiation.cc:769] Sending connection context
0926 07:29:53.891395 (+ 166us) client_negotiation.cc:240] Negotiation successful
0926 07:29:53.891664 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":150,"threads_started":1}
I20260926 07:29:53.893990 24962 heartbeater.cc:347] Connected to a master server at 127.22.53.254:38529
I20260926 07:29:53.894217 24962 heartbeater.cc:464] Registering TS with master...
I20260926 07:29:53.894647 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 24831
I20260926 07:29:53.894780 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407783529717-22743-0/minicluster-data/ts-2/wal/instance
I20260926 07:29:53.894922 24962 heartbeater.cc:511] Master 127.22.53.254:38529 requested a full tablet report, sending...
I20260926 07:29:53.895103 22743 external_mini_cluster.cc:1428] Setting key 41c9f10e87104a073db486d736b560a5
I20260926 07:29:53.896705 24479 ts_manager.cc:194] Registered new tserver with Master: 66291c8b70dc46a6acbfdf3c9c475500 (127.22.53.195:38915)
I20260926 07:29:53.897464 24479 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.53.195@KRBTEST.COM'} at 127.22.53.195:54731
I20260926 07:29:53.897688 22743 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:29:53.906410 24528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:53.896774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.195:53813 (local address 127.22.53.254:38529)
0926 07:29:53.897362 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:53.897367 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:53.897380 (+ 13us) server_negotiation.cc:407] Connection header received
0926 07:29:53.897416 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:53.897419 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:53.897462 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:53.897554 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:53.898238 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:53.899415 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:53.900731 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:53.900962 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:53.901057 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:53.902486 (+ 1429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:53.902490 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:53.902492 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:29:53.902518 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:53.904313 (+ 1795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:53.904911 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:53.904918 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:53.904923 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:53.905022 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:53.905373 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:53.905379 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:53.905381 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:53.905644 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:53.905768 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:53.906112 (+ 344us) server_negotiation.cc:299] Negotiation successful
0926 07:29:53.906219 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516}
I20260926 07:29:53.906543 24968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:53.896722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:38529 (local address 127.22.53.195:53813)
0926 07:29:53.897008 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:53.897020 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 07:29:53.897095 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:29:53.897615 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:29:53.897620 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:29:53.897931 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:29:53.898079 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:53.898089 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:53.899557 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:53.899560 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:53.900518 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:53.900567 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:53.901107 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:53.901112 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:53.901223 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:53.901788 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:29:53.901836 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:29:53.902330 (+ 494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:29:53.904471 (+ 2141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:29:53.904473 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:29:53.904475 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:29:53.904785 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:29:53.905150 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:29:53.905153 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:29:53.905154 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:29:53.905257 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:29:53.905762 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:29:53.905767 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:29:53.905920 (+ 153us) client_negotiation.cc:769] Sending connection context
0926 07:29:53.906036 (+ 116us) client_negotiation.cc:240] Negotiation successful
0926 07:29:53.906263 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":120,"threads_started":1}
Sep 26 07:29:53 dist-test-slave-wdjc krb5kdc[24434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407793, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 07:29:53 dist-test-slave-wdjc krb5kdc[24434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407793, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.53.254@KRBTEST.COM
I20260926 07:29:53.927385 24528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:53.918710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47056 (local address 127.22.53.254:38529)
0926 07:29:53.918879 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:53.918883 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:53.918896 (+ 13us) server_negotiation.cc:407] Connection header received
0926 07:29:53.918940 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:53.918942 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:53.918982 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:53.919082 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:53.919856 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:53.920397 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:53.921324 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:53.921553 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:53.921665 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:53.924191 (+ 2526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:53.924194 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:53.924196 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:29:53.924227 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:53.925612 (+ 1385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:53.926070 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:53.926076 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:53.926078 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:53.926171 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:53.926479 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:53.926482 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:53.926484 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:53.926684 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:53.926865 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:53.927085 (+ 220us) server_negotiation.cc:299] Negotiation successful
0926 07:29:53.927214 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 07:29:53.930678 24479 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47056:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:29:53.933885 24479 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:29:53.949936 24983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:53.944203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56258 (local address 127.22.53.193:38223)
0926 07:29:53.944886 (+ 683us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:53.944892 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:53.944911 (+ 19us) server_negotiation.cc:407] Connection header received
0926 07:29:53.944975 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:53.944979 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:53.945144 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:53.945311 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:29:53.945619 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:53.946822 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:53.948347 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:53.949087 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:53.949294 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:53.949472 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:53.949668 (+ 196us) server_negotiation.cc:299] Negotiation successful
0926 07:29:53.949781 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":92,"threads_started":1}
I20260926 07:29:53.949936 24982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:53.943685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.193:38223 (local address 127.0.0.1:56258)
0926 07:29:53.944558 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:53.944572 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:29:53.944659 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:29:53.945301 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:29:53.945307 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:29:53.945329 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:29:53.945487 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:53.945496 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:53.946952 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:53.946957 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:53.948216 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:53.948225 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:53.949299 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:53.949302 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:53.949425 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:53.949463 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 07:29:53.949618 (+ 155us) client_negotiation.cc:240] Negotiation successful
0926 07:29:53.949773 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":778,"mutex_wait_us":40,"thread_start_us":114,"threads_started":1}
I20260926 07:29:53.950274 24981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:53.944183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.194:41601 (local address 127.0.0.1:51260)
0926 07:29:53.944608 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:53.944623 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:29:53.944709 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:29:53.945504 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:29:53.945508 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:29:53.945537 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:29:53.945706 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:53.945714 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:53.947099 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:53.947104 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:53.948384 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:53.948395 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:53.949841 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:53.949845 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:53.949940 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:53.949964 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 07:29:53.950035 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 07:29:53.950111 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"mutex_wait_us":284,"spinlock_wait_cycles":5376,"thread_start_us":90,"threads_started":1}
I20260926 07:29:53.950618 24979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:53.943451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.195:38915 (local address 127.0.0.1:52908)
0926 07:29:53.943863 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:53.943915 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 07:29:53.944046 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:29:53.945504 (+ 1458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:29:53.945508 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:29:53.945538 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:29:53.945711 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:53.945716 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:53.947260 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:53.947263 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:53.949155 (+ 1892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:53.949165 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:53.950242 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:53.950246 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:53.950341 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:53.950365 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 07:29:53.950427 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 07:29:53.950489 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"mutex_wait_us":87,"spinlock_wait_cycles":4608,"thread_start_us":90,"threads_started":1}
I20260926 07:29:53.950731 24984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:53.944418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51260 (local address 127.22.53.194:41601)
0926 07:29:53.944905 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:53.944916 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:53.944940 (+ 24us) server_negotiation.cc:407] Connection header received
0926 07:29:53.945018 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:53.945023 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:53.945212 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:53.945361 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:29:53.945837 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:53.946975 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:53.948670 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:53.949532 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:53.949632 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:53.949778 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:53.950516 (+ 738us) server_negotiation.cc:299] Negotiation successful
0926 07:29:53.950605 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":114,"threads_started":1}
I20260926 07:29:53.950726 24980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:53.943657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52908 (local address 127.22.53.195:38915)
0926 07:29:53.944932 (+ 1275us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:53.944947 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:53.944967 (+ 20us) server_negotiation.cc:407] Connection header received
0926 07:29:53.945029 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:53.945033 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:53.945212 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:53.945360 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:29:53.945877 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:53.946643 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:53.949309 (+ 2666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:53.950114 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:53.950219 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:53.950431 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:53.950490 (+ 59us) server_negotiation.cc:299] Negotiation successful
0926 07:29:53.950584 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1182,"thread_start_us":128,"threads_started":1}
I20260926 07:29:53.953791 24616 tablet_service.cc:1467] Processing CreateTablet for tablet 7340e0073345498da8a3a776bf0913fd (DEFAULT_TABLE table=test-table [id=c951e9df9fc145479cd01464d14f1062]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:29:53.953799 24756 tablet_service.cc:1467] Processing CreateTablet for tablet 7340e0073345498da8a3a776bf0913fd (DEFAULT_TABLE table=test-table [id=c951e9df9fc145479cd01464d14f1062]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:29:53.953799 24896 tablet_service.cc:1467] Processing CreateTablet for tablet 7340e0073345498da8a3a776bf0913fd (DEFAULT_TABLE table=test-table [id=c951e9df9fc145479cd01464d14f1062]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:29:53.954994 24616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7340e0073345498da8a3a776bf0913fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:29:53.955003 24896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7340e0073345498da8a3a776bf0913fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:29:53.954995 24756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7340e0073345498da8a3a776bf0913fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:29:53.961822 24985 tablet_bootstrap.cc:492] T 7340e0073345498da8a3a776bf0913fd P 3a69b617c9f843b2a9001f6d598c63e8: Bootstrap starting.
I20260926 07:29:53.963784 24986 tablet_bootstrap.cc:492] T 7340e0073345498da8a3a776bf0913fd P ffbd74368f6e483abebb4ff75b582ba0: Bootstrap starting.
I20260926 07:29:53.963784 24987 tablet_bootstrap.cc:492] T 7340e0073345498da8a3a776bf0913fd P 66291c8b70dc46a6acbfdf3c9c475500: Bootstrap starting.
I20260926 07:29:53.964824 24985 tablet_bootstrap.cc:654] T 7340e0073345498da8a3a776bf0913fd P 3a69b617c9f843b2a9001f6d598c63e8: Neither blocks nor log segments found. Creating new log.
I20260926 07:29:53.965655 24985 log.cc:824] T 7340e0073345498da8a3a776bf0913fd P 3a69b617c9f843b2a9001f6d598c63e8: Log is configured to *not* fsync() on all Append() calls
I20260926 07:29:53.966292 24986 tablet_bootstrap.cc:654] T 7340e0073345498da8a3a776bf0913fd P ffbd74368f6e483abebb4ff75b582ba0: Neither blocks nor log segments found. Creating new log.
I20260926 07:29:53.966292 24987 tablet_bootstrap.cc:654] T 7340e0073345498da8a3a776bf0913fd P 66291c8b70dc46a6acbfdf3c9c475500: Neither blocks nor log segments found. Creating new log.
I20260926 07:29:53.967231 24986 log.cc:824] T 7340e0073345498da8a3a776bf0913fd P ffbd74368f6e483abebb4ff75b582ba0: Log is configured to *not* fsync() on all Append() calls
I20260926 07:29:53.967231 24987 log.cc:824] T 7340e0073345498da8a3a776bf0913fd P 66291c8b70dc46a6acbfdf3c9c475500: Log is configured to *not* fsync() on all Append() calls
I20260926 07:29:53.968796 24985 tablet_bootstrap.cc:492] T 7340e0073345498da8a3a776bf0913fd P 3a69b617c9f843b2a9001f6d598c63e8: No bootstrap required, opened a new log
I20260926 07:29:53.969038 24985 ts_tablet_manager.cc:1397] T 7340e0073345498da8a3a776bf0913fd P 3a69b617c9f843b2a9001f6d598c63e8: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260926 07:29:53.969465 24987 tablet_bootstrap.cc:492] T 7340e0073345498da8a3a776bf0913fd P 66291c8b70dc46a6acbfdf3c9c475500: No bootstrap required, opened a new log
I20260926 07:29:53.969465 24986 tablet_bootstrap.cc:492] T 7340e0073345498da8a3a776bf0913fd P ffbd74368f6e483abebb4ff75b582ba0: No bootstrap required, opened a new log
I20260926 07:29:53.969660 24986 ts_tablet_manager.cc:1397] T 7340e0073345498da8a3a776bf0913fd P ffbd74368f6e483abebb4ff75b582ba0: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 07:29:53.969660 24987 ts_tablet_manager.cc:1397] T 7340e0073345498da8a3a776bf0913fd P 66291c8b70dc46a6acbfdf3c9c475500: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 07:29:53.973191 24985 raft_consensus.cc:348] T 7340e0073345498da8a3a776bf0913fd P 3a69b617c9f843b2a9001f6d598c63e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66291c8b70dc46a6acbfdf3c9c475500" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 38915 } } peers { permanent_uuid: "ffbd74368f6e483abebb4ff75b582ba0" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 41601 } } peers { permanent_uuid: "3a69b617c9f843b2a9001f6d598c63e8" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 38223 } }
I20260926 07:29:53.973556 24985 raft_consensus.cc:374] T 7340e0073345498da8a3a776bf0913fd P 3a69b617c9f843b2a9001f6d598c63e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:29:53.973630 24985 raft_consensus.cc:729] T 7340e0073345498da8a3a776bf0913fd P 3a69b617c9f843b2a9001f6d598c63e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a69b617c9f843b2a9001f6d598c63e8, State: Initialized, Role: FOLLOWER
I20260926 07:29:53.974035 24987 raft_consensus.cc:348] T 7340e0073345498da8a3a776bf0913fd P 66291c8b70dc46a6acbfdf3c9c475500 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66291c8b70dc46a6acbfdf3c9c475500" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 38915 } } peers { permanent_uuid: "ffbd74368f6e483abebb4ff75b582ba0" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 41601 } } peers { permanent_uuid: "3a69b617c9f843b2a9001f6d598c63e8" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 38223 } }
I20260926 07:29:53.974035 24986 raft_consensus.cc:348] T 7340e0073345498da8a3a776bf0913fd P ffbd74368f6e483abebb4ff75b582ba0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66291c8b70dc46a6acbfdf3c9c475500" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 38915 } } peers { permanent_uuid: "ffbd74368f6e483abebb4ff75b582ba0" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 41601 } } peers { permanent_uuid: "3a69b617c9f843b2a9001f6d598c63e8" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 38223 } }
I20260926 07:29:53.974195 24985 consensus_queue.cc:260] T 7340e0073345498da8a3a776bf0913fd P 3a69b617c9f843b2a9001f6d598c63e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66291c8b70dc46a6acbfdf3c9c475500" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 38915 } } peers { permanent_uuid: "ffbd74368f6e483abebb4ff75b582ba0" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 41601 } } peers { permanent_uuid: "3a69b617c9f843b2a9001f6d598c63e8" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 38223 } }
I20260926 07:29:53.974310 24986 raft_consensus.cc:374] T 7340e0073345498da8a3a776bf0913fd P ffbd74368f6e483abebb4ff75b582ba0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:29:53.974310 24987 raft_consensus.cc:374] T 7340e0073345498da8a3a776bf0913fd P 66291c8b70dc46a6acbfdf3c9c475500 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:29:53.974362 24987 raft_consensus.cc:729] T 7340e0073345498da8a3a776bf0913fd P 66291c8b70dc46a6acbfdf3c9c475500 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66291c8b70dc46a6acbfdf3c9c475500, State: Initialized, Role: FOLLOWER
I20260926 07:29:53.974362 24986 raft_consensus.cc:729] T 7340e0073345498da8a3a776bf0913fd P ffbd74368f6e483abebb4ff75b582ba0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffbd74368f6e483abebb4ff75b582ba0, State: Initialized, Role: FOLLOWER
I20260926 07:29:53.974829 24987 consensus_queue.cc:260] T 7340e0073345498da8a3a776bf0913fd P 66291c8b70dc46a6acbfdf3c9c475500 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66291c8b70dc46a6acbfdf3c9c475500" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 38915 } } peers { permanent_uuid: "ffbd74368f6e483abebb4ff75b582ba0" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 41601 } } peers { permanent_uuid: "3a69b617c9f843b2a9001f6d598c63e8" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 38223 } }
I20260926 07:29:53.974829 24986 consensus_queue.cc:260] T 7340e0073345498da8a3a776bf0913fd P ffbd74368f6e483abebb4ff75b582ba0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66291c8b70dc46a6acbfdf3c9c475500" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 38915 } } peers { permanent_uuid: "ffbd74368f6e483abebb4ff75b582ba0" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 41601 } } peers { permanent_uuid: "3a69b617c9f843b2a9001f6d598c63e8" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 38223 } }
I20260926 07:29:53.975507 24682 heartbeater.cc:503] Master 127.22.53.254:38529 was elected leader, sending a full tablet report...
I20260926 07:29:53.975829 24987 ts_tablet_manager.cc:1428] T 7340e0073345498da8a3a776bf0913fd P 66291c8b70dc46a6acbfdf3c9c475500: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260926 07:29:53.975829 24986 ts_tablet_manager.cc:1428] T 7340e0073345498da8a3a776bf0913fd P ffbd74368f6e483abebb4ff75b582ba0: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260926 07:29:53.976177 24962 heartbeater.cc:503] Master 127.22.53.254:38529 was elected leader, sending a full tablet report...
I20260926 07:29:53.976221 24822 heartbeater.cc:503] Master 127.22.53.254:38529 was elected leader, sending a full tablet report...
I20260926 07:29:53.977172 24985 ts_tablet_manager.cc:1428] T 7340e0073345498da8a3a776bf0913fd P 3a69b617c9f843b2a9001f6d598c63e8: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260926 07:29:54.129611 24993 raft_consensus.cc:482] T 7340e0073345498da8a3a776bf0913fd P 66291c8b70dc46a6acbfdf3c9c475500 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:29:54.129966 24993 raft_consensus.cc:504] T 7340e0073345498da8a3a776bf0913fd P 66291c8b70dc46a6acbfdf3c9c475500 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66291c8b70dc46a6acbfdf3c9c475500" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 38915 } } peers { permanent_uuid: "ffbd74368f6e483abebb4ff75b582ba0" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 41601 } } peers { permanent_uuid: "3a69b617c9f843b2a9001f6d598c63e8" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 38223 } }
I20260926 07:29:54.131426 24993 leader_election.cc:288] T 7340e0073345498da8a3a776bf0913fd P 66291c8b70dc46a6acbfdf3c9c475500 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ffbd74368f6e483abebb4ff75b582ba0 (127.22.53.194:41601), 3a69b617c9f843b2a9001f6d598c63e8 (127.22.53.193:38223)
I20260926 07:29:54.137123 24984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:54.132017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.195:44827 (local address 127.22.53.194:41601)
0926 07:29:54.132206 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:54.132212 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:54.132237 (+ 25us) server_negotiation.cc:407] Connection header received
0926 07:29:54.132413 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:54.132417 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:54.132482 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:54.132608 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:29:54.133107 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:54.133812 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:54.135420 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:54.136106 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:54.136236 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:54.136286 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:54.136856 (+ 570us) server_negotiation.cc:299] Negotiation successful
0926 07:29:54.136950 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260926 07:29:54.137171 24855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:54.131687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.194:41601 (local address 127.22.53.195:44827)
0926 07:29:54.132040 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:54.132061 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 07:29:54.132224 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:29:54.132671 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:29:54.132674 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:29:54.132702 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:29:54.132933 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:54.132940 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:54.133955 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:54.133958 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:54.135261 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:54.135273 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:54.136261 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:54.136265 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:54.136605 (+ 340us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:54.136648 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 07:29:54.136787 (+ 139us) client_negotiation.cc:240] Negotiation successful
0926 07:29:54.136929 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"mutex_wait_us":43}
I20260926 07:29:54.137756 24994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:54.132056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.193:38223 (local address 127.22.53.195:54239)
0926 07:29:54.132677 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:54.132691 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:29:54.132792 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:29:54.133046 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:29:54.133051 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:29:54.133074 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:29:54.133605 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:54.133611 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:54.134563 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:54.134568 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:54.135721 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:54.135734 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:54.137163 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:54.137170 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:54.137336 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:54.137374 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 07:29:54.137493 (+ 119us) client_negotiation.cc:240] Negotiation successful
0926 07:29:54.137589 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"mutex_wait_us":27,"thread_start_us":212,"threads_started":1}
I20260926 07:29:54.137774 24983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:54.132197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.195:54239 (local address 127.22.53.193:38223)
0926 07:29:54.132366 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:54.132372 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:54.132803 (+ 431us) server_negotiation.cc:407] Connection header received
0926 07:29:54.132876 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:54.132878 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:54.132921 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:54.132998 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:29:54.133765 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:54.134418 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:54.135927 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:54.136967 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:54.137126 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:54.137195 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:54.137536 (+ 341us) server_negotiation.cc:299] Negotiation successful
0926 07:29:54.137641 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260926 07:29:54.137959 24823 tablet.cc:2369] T 7340e0073345498da8a3a776bf0913fd P ffbd74368f6e483abebb4ff75b582ba0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:29:54.138099 24776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7340e0073345498da8a3a776bf0913fd" candidate_uuid: "66291c8b70dc46a6acbfdf3c9c475500" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffbd74368f6e483abebb4ff75b582ba0" is_pre_election: true
I20260926 07:29:54.138422 24776 raft_consensus.cc:2454] T 7340e0073345498da8a3a776bf0913fd P ffbd74368f6e483abebb4ff75b582ba0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66291c8b70dc46a6acbfdf3c9c475500 in term 0.
I20260926 07:29:54.138705 24636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7340e0073345498da8a3a776bf0913fd" candidate_uuid: "66291c8b70dc46a6acbfdf3c9c475500" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a69b617c9f843b2a9001f6d598c63e8" is_pre_election: true
I20260926 07:29:54.139039 24636 raft_consensus.cc:2454] T 7340e0073345498da8a3a776bf0913fd P 3a69b617c9f843b2a9001f6d598c63e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66291c8b70dc46a6acbfdf3c9c475500 in term 0.
I20260926 07:29:54.139040 24847 leader_election.cc:302] T 7340e0073345498da8a3a776bf0913fd P 66291c8b70dc46a6acbfdf3c9c475500 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66291c8b70dc46a6acbfdf3c9c475500, ffbd74368f6e483abebb4ff75b582ba0; no voters:
I20260926 07:29:54.139402 24993 raft_consensus.cc:2781] T 7340e0073345498da8a3a776bf0913fd P 66291c8b70dc46a6acbfdf3c9c475500 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:29:54.139515 24993 raft_consensus.cc:482] T 7340e0073345498da8a3a776bf0913fd P 66291c8b70dc46a6acbfdf3c9c475500 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:29:54.139582 24993 raft_consensus.cc:3037] T 7340e0073345498da8a3a776bf0913fd P 66291c8b70dc46a6acbfdf3c9c475500 [term 0 FOLLOWER]: Advancing to term 1
W20260926 07:29:54.140758 24963 tablet.cc:2369] T 7340e0073345498da8a3a776bf0913fd P 66291c8b70dc46a6acbfdf3c9c475500: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:29:54.140970 24993 raft_consensus.cc:504] T 7340e0073345498da8a3a776bf0913fd P 66291c8b70dc46a6acbfdf3c9c475500 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66291c8b70dc46a6acbfdf3c9c475500" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 38915 } } peers { permanent_uuid: "ffbd74368f6e483abebb4ff75b582ba0" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 41601 } } peers { permanent_uuid: "3a69b617c9f843b2a9001f6d598c63e8" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 38223 } }
I20260926 07:29:54.141427 24993 leader_election.cc:288] T 7340e0073345498da8a3a776bf0913fd P 66291c8b70dc46a6acbfdf3c9c475500 [CANDIDATE]: Term 1 election: Requested vote from peers ffbd74368f6e483abebb4ff75b582ba0 (127.22.53.194:41601), 3a69b617c9f843b2a9001f6d598c63e8 (127.22.53.193:38223)
I20260926 07:29:54.141949 24636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7340e0073345498da8a3a776bf0913fd" candidate_uuid: "66291c8b70dc46a6acbfdf3c9c475500" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a69b617c9f843b2a9001f6d598c63e8"
I20260926 07:29:54.142127 24636 raft_consensus.cc:3037] T 7340e0073345498da8a3a776bf0913fd P 3a69b617c9f843b2a9001f6d598c63e8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:29:54.143404 24636 raft_consensus.cc:2454] T 7340e0073345498da8a3a776bf0913fd P 3a69b617c9f843b2a9001f6d598c63e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66291c8b70dc46a6acbfdf3c9c475500 in term 1.
I20260926 07:29:54.143437 24776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7340e0073345498da8a3a776bf0913fd" candidate_uuid: "66291c8b70dc46a6acbfdf3c9c475500" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffbd74368f6e483abebb4ff75b582ba0"
I20260926 07:29:54.143733 24776 raft_consensus.cc:3037] T 7340e0073345498da8a3a776bf0913fd P ffbd74368f6e483abebb4ff75b582ba0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:29:54.143857 24849 leader_election.cc:302] T 7340e0073345498da8a3a776bf0913fd P 66291c8b70dc46a6acbfdf3c9c475500 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a69b617c9f843b2a9001f6d598c63e8, 66291c8b70dc46a6acbfdf3c9c475500; no voters:
I20260926 07:29:54.144115 24993 raft_consensus.cc:2781] T 7340e0073345498da8a3a776bf0913fd P 66291c8b70dc46a6acbfdf3c9c475500 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:29:54.144411 24993 raft_consensus.cc:686] T 7340e0073345498da8a3a776bf0913fd P 66291c8b70dc46a6acbfdf3c9c475500 [term 1 LEADER]: Becoming Leader. State: Replica: 66291c8b70dc46a6acbfdf3c9c475500, State: Running, Role: LEADER
I20260926 07:29:54.144855 24993 consensus_queue.cc:237] T 7340e0073345498da8a3a776bf0913fd P 66291c8b70dc46a6acbfdf3c9c475500 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66291c8b70dc46a6acbfdf3c9c475500" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 38915 } } peers { permanent_uuid: "ffbd74368f6e483abebb4ff75b582ba0" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 41601 } } peers { permanent_uuid: "3a69b617c9f843b2a9001f6d598c63e8" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 38223 } }
I20260926 07:29:54.145394 24776 raft_consensus.cc:2454] T 7340e0073345498da8a3a776bf0913fd P ffbd74368f6e483abebb4ff75b582ba0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66291c8b70dc46a6acbfdf3c9c475500 in term 1.
I20260926 07:29:54.149339 24479 catalog_manager.cc:5495] T 7340e0073345498da8a3a776bf0913fd P 66291c8b70dc46a6acbfdf3c9c475500 reported cstate change: term changed from 0 to 1, leader changed from <none> to 66291c8b70dc46a6acbfdf3c9c475500 (127.22.53.195). New cstate: current_term: 1 leader_uuid: "66291c8b70dc46a6acbfdf3c9c475500" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66291c8b70dc46a6acbfdf3c9c475500" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 38915 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ffbd74368f6e483abebb4ff75b582ba0" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 41601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a69b617c9f843b2a9001f6d598c63e8" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 38223 } health_report { overall_health: UNKNOWN } } }
W20260926 07:29:54.153195 24683 tablet.cc:2369] T 7340e0073345498da8a3a776bf0913fd P 3a69b617c9f843b2a9001f6d598c63e8: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 26 07:29:54 dist-test-slave-wdjc krb5kdc[24434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407793, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.254@KRBTEST.COM for kudu/127.22.53.254@KRBTEST.COM
I20260926 07:29:54.221129 25002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:54.210862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:38529 (local address 127.0.0.1:47058)
0926 07:29:54.211407 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:54.211422 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:29:54.211806 (+ 384us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:29:54.212094 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:29:54.212097 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:29:54.212696 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:29:54.212811 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:54.212819 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:54.213631 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:54.213638 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:54.214200 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:54.214209 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:54.214680 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:54.214683 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:54.214764 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:54.215476 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:29:54.215490 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:29:54.217569 (+ 2079us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:29:54.219371 (+ 1802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:29:54.219389 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:29:54.219397 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:29:54.219670 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:29:54.219964 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:29:54.219967 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:29:54.219969 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:29:54.220052 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:29:54.220470 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:29:54.220479 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:29:54.220648 (+ 169us) client_negotiation.cc:769] Sending connection context
0926 07:29:54.220819 (+ 171us) client_negotiation.cc:240] Negotiation successful
0926 07:29:54.220979 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":169,"threads_started":1}
I20260926 07:29:54.221237 24528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:54.210913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47058 (local address 127.22.53.254:38529)
0926 07:29:54.211265 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:54.211272 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:54.211691 (+ 419us) server_negotiation.cc:407] Connection header received
0926 07:29:54.211924 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:54.211929 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:54.211978 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:54.212070 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:54.212943 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:54.213509 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:54.214323 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:54.214556 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:54.214670 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:54.217757 (+ 3087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:54.217760 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:54.217762 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:29:54.217789 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:54.219224 (+ 1435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:54.219777 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:54.219781 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:54.219784 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:54.219865 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:54.220138 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:54.220144 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:54.220146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:54.220352 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:54.220461 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:54.220943 (+ 482us) server_negotiation.cc:299] Negotiation successful
0926 07:29:54.221076 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":108}
I20260926 07:29:54.229964 24479 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.22.53.254@KRBTEST.COM'} at 127.0.0.1:47058:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 07:29:54.231711 24479 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:29:54.238945 24616 tablet_service.cc:1467] Processing CreateTablet for tablet e6bd5365020f400d8946803e5cdf5390 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6ca2df0e22a74d54a652bc36b06e058f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 07:29:54.238955 24896 tablet_service.cc:1467] Processing CreateTablet for tablet e6bd5365020f400d8946803e5cdf5390 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6ca2df0e22a74d54a652bc36b06e058f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 07:29:54.239851 24896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6bd5365020f400d8946803e5cdf5390. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:29:54.239850 24616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6bd5365020f400d8946803e5cdf5390. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:29:54.242120 24756 tablet_service.cc:1467] Processing CreateTablet for tablet e6bd5365020f400d8946803e5cdf5390 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6ca2df0e22a74d54a652bc36b06e058f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 07:29:54.243070 24756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6bd5365020f400d8946803e5cdf5390. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:29:54.243379 24987 tablet_bootstrap.cc:492] T e6bd5365020f400d8946803e5cdf5390 P 66291c8b70dc46a6acbfdf3c9c475500: Bootstrap starting.
I20260926 07:29:54.244666 24987 tablet_bootstrap.cc:654] T e6bd5365020f400d8946803e5cdf5390 P 66291c8b70dc46a6acbfdf3c9c475500: Neither blocks nor log segments found. Creating new log.
I20260926 07:29:54.246213 24987 tablet_bootstrap.cc:492] T e6bd5365020f400d8946803e5cdf5390 P 66291c8b70dc46a6acbfdf3c9c475500: No bootstrap required, opened a new log
I20260926 07:29:54.246369 24987 ts_tablet_manager.cc:1397] T e6bd5365020f400d8946803e5cdf5390 P 66291c8b70dc46a6acbfdf3c9c475500: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260926 07:29:54.246428 24986 tablet_bootstrap.cc:492] T e6bd5365020f400d8946803e5cdf5390 P ffbd74368f6e483abebb4ff75b582ba0: Bootstrap starting.
I20260926 07:29:54.246897 24987 raft_consensus.cc:348] T e6bd5365020f400d8946803e5cdf5390 P 66291c8b70dc46a6acbfdf3c9c475500 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a69b617c9f843b2a9001f6d598c63e8" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 38223 } } peers { permanent_uuid: "ffbd74368f6e483abebb4ff75b582ba0" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 41601 } } peers { permanent_uuid: "66291c8b70dc46a6acbfdf3c9c475500" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 38915 } }
I20260926 07:29:54.247051 24987 raft_consensus.cc:374] T e6bd5365020f400d8946803e5cdf5390 P 66291c8b70dc46a6acbfdf3c9c475500 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:29:54.247083 24987 raft_consensus.cc:729] T e6bd5365020f400d8946803e5cdf5390 P 66291c8b70dc46a6acbfdf3c9c475500 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66291c8b70dc46a6acbfdf3c9c475500, State: Initialized, Role: FOLLOWER
I20260926 07:29:54.247258 24987 consensus_queue.cc:260] T e6bd5365020f400d8946803e5cdf5390 P 66291c8b70dc46a6acbfdf3c9c475500 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a69b617c9f843b2a9001f6d598c63e8" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 38223 } } peers { permanent_uuid: "ffbd74368f6e483abebb4ff75b582ba0" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 41601 } } peers { permanent_uuid: "66291c8b70dc46a6acbfdf3c9c475500" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 38915 } }
I20260926 07:29:54.247570 24986 tablet_bootstrap.cc:654] T e6bd5365020f400d8946803e5cdf5390 P ffbd74368f6e483abebb4ff75b582ba0: Neither blocks nor log segments found. Creating new log.
I20260926 07:29:54.247601 24987 ts_tablet_manager.cc:1428] T e6bd5365020f400d8946803e5cdf5390 P 66291c8b70dc46a6acbfdf3c9c475500: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260926 07:29:54.247550 24993 tablet_replica.cc:439] T e6bd5365020f400d8946803e5cdf5390 P 66291c8b70dc46a6acbfdf3c9c475500: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a69b617c9f843b2a9001f6d598c63e8" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 38223 } } peers { permanent_uuid: "ffbd74368f6e483abebb4ff75b582ba0" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 41601 } } peers { permanent_uuid: "66291c8b70dc46a6acbfdf3c9c475500" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 38915 } } }
I20260926 07:29:54.247687 24993 tablet_replica.cc:442] T e6bd5365020f400d8946803e5cdf5390 P 66291c8b70dc46a6acbfdf3c9c475500: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:29:54.249185 24986 tablet_bootstrap.cc:492] T e6bd5365020f400d8946803e5cdf5390 P ffbd74368f6e483abebb4ff75b582ba0: No bootstrap required, opened a new log
I20260926 07:29:54.249259 24985 tablet_bootstrap.cc:492] T e6bd5365020f400d8946803e5cdf5390 P 3a69b617c9f843b2a9001f6d598c63e8: Bootstrap starting.
I20260926 07:29:54.249287 24986 ts_tablet_manager.cc:1397] T e6bd5365020f400d8946803e5cdf5390 P ffbd74368f6e483abebb4ff75b582ba0: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 07:29:54.249773 24986 raft_consensus.cc:348] T e6bd5365020f400d8946803e5cdf5390 P ffbd74368f6e483abebb4ff75b582ba0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a69b617c9f843b2a9001f6d598c63e8" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 38223 } } peers { permanent_uuid: "ffbd74368f6e483abebb4ff75b582ba0" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 41601 } } peers { permanent_uuid: "66291c8b70dc46a6acbfdf3c9c475500" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 38915 } }
I20260926 07:29:54.249912 24986 raft_consensus.cc:374] T e6bd5365020f400d8946803e5cdf5390 P ffbd74368f6e483abebb4ff75b582ba0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:29:54.249955 24986 raft_consensus.cc:729] T e6bd5365020f400d8946803e5cdf5390 P ffbd74368f6e483abebb4ff75b582ba0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffbd74368f6e483abebb4ff75b582ba0, State: Initialized, Role: FOLLOWER
I20260926 07:29:54.250145 24986 consensus_queue.cc:260] T e6bd5365020f400d8946803e5cdf5390 P ffbd74368f6e483abebb4ff75b582ba0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a69b617c9f843b2a9001f6d598c63e8" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 38223 } } peers { permanent_uuid: "ffbd74368f6e483abebb4ff75b582ba0" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 41601 } } peers { permanent_uuid: "66291c8b70dc46a6acbfdf3c9c475500" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 38915 } }
I20260926 07:29:54.250444 24985 tablet_bootstrap.cc:654] T e6bd5365020f400d8946803e5cdf5390 P 3a69b617c9f843b2a9001f6d598c63e8: Neither blocks nor log segments found. Creating new log.
I20260926 07:29:54.250443 24992 tablet_replica.cc:439] T e6bd5365020f400d8946803e5cdf5390 P ffbd74368f6e483abebb4ff75b582ba0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a69b617c9f843b2a9001f6d598c63e8" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 38223 } } peers { permanent_uuid: "ffbd74368f6e483abebb4ff75b582ba0" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 41601 } } peers { permanent_uuid: "66291c8b70dc46a6acbfdf3c9c475500" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 38915 } } }
I20260926 07:29:54.250590 24992 tablet_replica.cc:442] T e6bd5365020f400d8946803e5cdf5390 P ffbd74368f6e483abebb4ff75b582ba0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:29:54.250866 24986 ts_tablet_manager.cc:1428] T e6bd5365020f400d8946803e5cdf5390 P ffbd74368f6e483abebb4ff75b582ba0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 07:29:54.252257 24985 tablet_bootstrap.cc:492] T e6bd5365020f400d8946803e5cdf5390 P 3a69b617c9f843b2a9001f6d598c63e8: No bootstrap required, opened a new log
I20260926 07:29:54.252360 24985 ts_tablet_manager.cc:1397] T e6bd5365020f400d8946803e5cdf5390 P 3a69b617c9f843b2a9001f6d598c63e8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 07:29:54.253057 24985 raft_consensus.cc:348] T e6bd5365020f400d8946803e5cdf5390 P 3a69b617c9f843b2a9001f6d598c63e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a69b617c9f843b2a9001f6d598c63e8" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 38223 } } peers { permanent_uuid: "ffbd74368f6e483abebb4ff75b582ba0" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 41601 } } peers { permanent_uuid: "66291c8b70dc46a6acbfdf3c9c475500" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 38915 } }
I20260926 07:29:54.253171 24985 raft_consensus.cc:374] T e6bd5365020f400d8946803e5cdf5390 P 3a69b617c9f843b2a9001f6d598c63e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:29:54.253194 24985 raft_consensus.cc:729] T e6bd5365020f400d8946803e5cdf5390 P 3a69b617c9f843b2a9001f6d598c63e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a69b617c9f843b2a9001f6d598c63e8, State: Initialized, Role: FOLLOWER
I20260926 07:29:54.253332 24985 consensus_queue.cc:260] T e6bd5365020f400d8946803e5cdf5390 P 3a69b617c9f843b2a9001f6d598c63e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a69b617c9f843b2a9001f6d598c63e8" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 38223 } } peers { permanent_uuid: "ffbd74368f6e483abebb4ff75b582ba0" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 41601 } } peers { permanent_uuid: "66291c8b70dc46a6acbfdf3c9c475500" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 38915 } }
I20260926 07:29:54.253639 24985 ts_tablet_manager.cc:1428] T e6bd5365020f400d8946803e5cdf5390 P 3a69b617c9f843b2a9001f6d598c63e8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260926 07:29:54.253615 24991 tablet_replica.cc:439] T e6bd5365020f400d8946803e5cdf5390 P 3a69b617c9f843b2a9001f6d598c63e8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a69b617c9f843b2a9001f6d598c63e8" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 38223 } } peers { permanent_uuid: "ffbd74368f6e483abebb4ff75b582ba0" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 41601 } } peers { permanent_uuid: "66291c8b70dc46a6acbfdf3c9c475500" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 38915 } } }
I20260926 07:29:54.253790 24991 tablet_replica.cc:442] T e6bd5365020f400d8946803e5cdf5390 P 3a69b617c9f843b2a9001f6d598c63e8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:29:54.369721 24993 raft_consensus.cc:482] T e6bd5365020f400d8946803e5cdf5390 P 66291c8b70dc46a6acbfdf3c9c475500 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:29:54.369863 24993 raft_consensus.cc:504] T e6bd5365020f400d8946803e5cdf5390 P 66291c8b70dc46a6acbfdf3c9c475500 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a69b617c9f843b2a9001f6d598c63e8" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 38223 } } peers { permanent_uuid: "ffbd74368f6e483abebb4ff75b582ba0" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 41601 } } peers { permanent_uuid: "66291c8b70dc46a6acbfdf3c9c475500" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 38915 } }
I20260926 07:29:54.370254 24993 leader_election.cc:288] T e6bd5365020f400d8946803e5cdf5390 P 66291c8b70dc46a6acbfdf3c9c475500 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3a69b617c9f843b2a9001f6d598c63e8 (127.22.53.193:38223), ffbd74368f6e483abebb4ff75b582ba0 (127.22.53.194:41601)
I20260926 07:29:54.370774 24776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6bd5365020f400d8946803e5cdf5390" candidate_uuid: "66291c8b70dc46a6acbfdf3c9c475500" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffbd74368f6e483abebb4ff75b582ba0" is_pre_election: true
I20260926 07:29:54.370823 24636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6bd5365020f400d8946803e5cdf5390" candidate_uuid: "66291c8b70dc46a6acbfdf3c9c475500" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a69b617c9f843b2a9001f6d598c63e8" is_pre_election: true
I20260926 07:29:54.370918 24776 raft_consensus.cc:2454] T e6bd5365020f400d8946803e5cdf5390 P ffbd74368f6e483abebb4ff75b582ba0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66291c8b70dc46a6acbfdf3c9c475500 in term 0.
I20260926 07:29:54.371048 24636 raft_consensus.cc:2454] T e6bd5365020f400d8946803e5cdf5390 P 3a69b617c9f843b2a9001f6d598c63e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66291c8b70dc46a6acbfdf3c9c475500 in term 0.
I20260926 07:29:54.371323 24847 leader_election.cc:302] T e6bd5365020f400d8946803e5cdf5390 P 66291c8b70dc46a6acbfdf3c9c475500 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66291c8b70dc46a6acbfdf3c9c475500, ffbd74368f6e483abebb4ff75b582ba0; no voters:
I20260926 07:29:54.371508 24993 raft_consensus.cc:2781] T e6bd5365020f400d8946803e5cdf5390 P 66291c8b70dc46a6acbfdf3c9c475500 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:29:54.371569 24993 raft_consensus.cc:482] T e6bd5365020f400d8946803e5cdf5390 P 66291c8b70dc46a6acbfdf3c9c475500 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:29:54.371614 24993 raft_consensus.cc:3037] T e6bd5365020f400d8946803e5cdf5390 P 66291c8b70dc46a6acbfdf3c9c475500 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:29:54.372681 24993 raft_consensus.cc:504] T e6bd5365020f400d8946803e5cdf5390 P 66291c8b70dc46a6acbfdf3c9c475500 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a69b617c9f843b2a9001f6d598c63e8" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 38223 } } peers { permanent_uuid: "ffbd74368f6e483abebb4ff75b582ba0" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 41601 } } peers { permanent_uuid: "66291c8b70dc46a6acbfdf3c9c475500" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 38915 } }
I20260926 07:29:54.373047 24993 leader_election.cc:288] T e6bd5365020f400d8946803e5cdf5390 P 66291c8b70dc46a6acbfdf3c9c475500 [CANDIDATE]: Term 1 election: Requested vote from peers 3a69b617c9f843b2a9001f6d598c63e8 (127.22.53.193:38223), ffbd74368f6e483abebb4ff75b582ba0 (127.22.53.194:41601)
I20260926 07:29:54.373476 24776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6bd5365020f400d8946803e5cdf5390" candidate_uuid: "66291c8b70dc46a6acbfdf3c9c475500" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffbd74368f6e483abebb4ff75b582ba0"
I20260926 07:29:54.373468 24636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6bd5365020f400d8946803e5cdf5390" candidate_uuid: "66291c8b70dc46a6acbfdf3c9c475500" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a69b617c9f843b2a9001f6d598c63e8"
I20260926 07:29:54.373633 24776 raft_consensus.cc:3037] T e6bd5365020f400d8946803e5cdf5390 P ffbd74368f6e483abebb4ff75b582ba0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:29:54.373644 24636 raft_consensus.cc:3037] T e6bd5365020f400d8946803e5cdf5390 P 3a69b617c9f843b2a9001f6d598c63e8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:29:54.374634 24776 raft_consensus.cc:2454] T e6bd5365020f400d8946803e5cdf5390 P ffbd74368f6e483abebb4ff75b582ba0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66291c8b70dc46a6acbfdf3c9c475500 in term 1.
I20260926 07:29:54.374636 24636 raft_consensus.cc:2454] T e6bd5365020f400d8946803e5cdf5390 P 3a69b617c9f843b2a9001f6d598c63e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66291c8b70dc46a6acbfdf3c9c475500 in term 1.
I20260926 07:29:54.375010 24849 leader_election.cc:302] T e6bd5365020f400d8946803e5cdf5390 P 66291c8b70dc46a6acbfdf3c9c475500 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a69b617c9f843b2a9001f6d598c63e8, 66291c8b70dc46a6acbfdf3c9c475500; no voters:
I20260926 07:29:54.375222 24993 raft_consensus.cc:2781] T e6bd5365020f400d8946803e5cdf5390 P 66291c8b70dc46a6acbfdf3c9c475500 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:29:54.375339 24993 raft_consensus.cc:686] T e6bd5365020f400d8946803e5cdf5390 P 66291c8b70dc46a6acbfdf3c9c475500 [term 1 LEADER]: Becoming Leader. State: Replica: 66291c8b70dc46a6acbfdf3c9c475500, State: Running, Role: LEADER
I20260926 07:29:54.375491 24993 consensus_queue.cc:237] T e6bd5365020f400d8946803e5cdf5390 P 66291c8b70dc46a6acbfdf3c9c475500 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a69b617c9f843b2a9001f6d598c63e8" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 38223 } } peers { permanent_uuid: "ffbd74368f6e483abebb4ff75b582ba0" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 41601 } } peers { permanent_uuid: "66291c8b70dc46a6acbfdf3c9c475500" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 38915 } }
I20260926 07:29:54.376196 24995 tablet_replica.cc:439] T e6bd5365020f400d8946803e5cdf5390 P 66291c8b70dc46a6acbfdf3c9c475500: TxnStatusTablet state changed. Reason: New leader 66291c8b70dc46a6acbfdf3c9c475500. Latest consensus state: current_term: 1 leader_uuid: "66291c8b70dc46a6acbfdf3c9c475500" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a69b617c9f843b2a9001f6d598c63e8" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 38223 } } peers { permanent_uuid: "ffbd74368f6e483abebb4ff75b582ba0" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 41601 } } peers { permanent_uuid: "66291c8b70dc46a6acbfdf3c9c475500" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 38915 } } }
I20260926 07:29:54.376322 24995 tablet_replica.cc:442] T e6bd5365020f400d8946803e5cdf5390 P 66291c8b70dc46a6acbfdf3c9c475500: This TxnStatusTablet replica's current role is: LEADER
I20260926 07:29:54.376659 25007 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 07:29:54.376924 25007 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 07:29:54.377406 24479 catalog_manager.cc:5495] T e6bd5365020f400d8946803e5cdf5390 P 66291c8b70dc46a6acbfdf3c9c475500 reported cstate change: term changed from 0 to 1, leader changed from <none> to 66291c8b70dc46a6acbfdf3c9c475500 (127.22.53.195). New cstate: current_term: 1 leader_uuid: "66291c8b70dc46a6acbfdf3c9c475500" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a69b617c9f843b2a9001f6d598c63e8" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 38223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ffbd74368f6e483abebb4ff75b582ba0" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 41601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66291c8b70dc46a6acbfdf3c9c475500" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 38915 } health_report { overall_health: HEALTHY } } }
I20260926 07:29:54.409060 24980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:54.404227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52916 (local address 127.22.53.195:38915)
0926 07:29:54.404385 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:54.404390 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:54.404406 (+ 16us) server_negotiation.cc:407] Connection header received
0926 07:29:54.404490 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:54.404493 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:54.404588 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:54.404705 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:29:54.404988 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:54.405822 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:54.407119 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:54.407476 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:54.407583 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:54.407936 (+ 353us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:29:54.408609 (+ 673us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:29:54.408727 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:54.408824 (+ 97us) server_negotiation.cc:299] Negotiation successful
0926 07:29:54.408895 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 07:29:54.409087 25002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:54.403887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.195:38915 (local address 127.0.0.1:52916)
0926 07:29:54.404041 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:54.404056 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:29:54.404167 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:29:54.404733 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:29:54.404737 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:29:54.404745 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 07:29:54.404868 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:54.404874 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:54.405974 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:54.405978 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:54.406984 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:54.406997 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:54.407608 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:54.407611 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:54.407710 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:54.407781 (+ 71us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 07:29:54.408717 (+ 936us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 07:29:54.408735 (+ 18us) client_negotiation.cc:769] Sending connection context
0926 07:29:54.408816 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 07:29:54.408933 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260926 07:29:54.416014 24776 raft_consensus.cc:1261] T e6bd5365020f400d8946803e5cdf5390 P ffbd74368f6e483abebb4ff75b582ba0 [term 1 FOLLOWER]: Refusing update from remote peer 66291c8b70dc46a6acbfdf3c9c475500: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:29:54.416019 24636 raft_consensus.cc:1261] T e6bd5365020f400d8946803e5cdf5390 P 3a69b617c9f843b2a9001f6d598c63e8 [term 1 FOLLOWER]: Refusing update from remote peer 66291c8b70dc46a6acbfdf3c9c475500: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:29:54.417009 24995 consensus_queue.cc:1035] T e6bd5365020f400d8946803e5cdf5390 P 66291c8b70dc46a6acbfdf3c9c475500 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a69b617c9f843b2a9001f6d598c63e8" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 38223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:29:54.417788 24995 consensus_queue.cc:1035] T e6bd5365020f400d8946803e5cdf5390 P 66291c8b70dc46a6acbfdf3c9c475500 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ffbd74368f6e483abebb4ff75b582ba0" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 41601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:29:54.422432 24992 tablet_replica.cc:439] T e6bd5365020f400d8946803e5cdf5390 P ffbd74368f6e483abebb4ff75b582ba0: TxnStatusTablet state changed. Reason: New leader 66291c8b70dc46a6acbfdf3c9c475500. Latest consensus state: current_term: 1 leader_uuid: "66291c8b70dc46a6acbfdf3c9c475500" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a69b617c9f843b2a9001f6d598c63e8" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 38223 } } peers { permanent_uuid: "ffbd74368f6e483abebb4ff75b582ba0" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 41601 } } peers { permanent_uuid: "66291c8b70dc46a6acbfdf3c9c475500" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 38915 } } }
I20260926 07:29:54.422607 24992 tablet_replica.cc:442] T e6bd5365020f400d8946803e5cdf5390 P ffbd74368f6e483abebb4ff75b582ba0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:29:54.422600 24991 tablet_replica.cc:439] T e6bd5365020f400d8946803e5cdf5390 P 3a69b617c9f843b2a9001f6d598c63e8: TxnStatusTablet state changed. Reason: New leader 66291c8b70dc46a6acbfdf3c9c475500. Latest consensus state: current_term: 1 leader_uuid: "66291c8b70dc46a6acbfdf3c9c475500" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a69b617c9f843b2a9001f6d598c63e8" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 38223 } } peers { permanent_uuid: "ffbd74368f6e483abebb4ff75b582ba0" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 41601 } } peers { permanent_uuid: "66291c8b70dc46a6acbfdf3c9c475500" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 38915 } } }
I20260926 07:29:54.422725 24991 tablet_replica.cc:442] T e6bd5365020f400d8946803e5cdf5390 P 3a69b617c9f843b2a9001f6d598c63e8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:29:54.426635 24993 tablet_replica.cc:439] T e6bd5365020f400d8946803e5cdf5390 P 66291c8b70dc46a6acbfdf3c9c475500: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "66291c8b70dc46a6acbfdf3c9c475500" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a69b617c9f843b2a9001f6d598c63e8" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 38223 } } peers { permanent_uuid: "ffbd74368f6e483abebb4ff75b582ba0" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 41601 } } peers { permanent_uuid: "66291c8b70dc46a6acbfdf3c9c475500" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 38915 } } }
I20260926 07:29:54.426793 24993 tablet_replica.cc:442] T e6bd5365020f400d8946803e5cdf5390 P 66291c8b70dc46a6acbfdf3c9c475500: This TxnStatusTablet replica's current role is: LEADER
I20260926 07:29:54.426903 24992 tablet_replica.cc:439] T e6bd5365020f400d8946803e5cdf5390 P ffbd74368f6e483abebb4ff75b582ba0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "66291c8b70dc46a6acbfdf3c9c475500" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a69b617c9f843b2a9001f6d598c63e8" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 38223 } } peers { permanent_uuid: "ffbd74368f6e483abebb4ff75b582ba0" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 41601 } } peers { permanent_uuid: "66291c8b70dc46a6acbfdf3c9c475500" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 38915 } } }
I20260926 07:29:54.427013 24992 tablet_replica.cc:442] T e6bd5365020f400d8946803e5cdf5390 P ffbd74368f6e483abebb4ff75b582ba0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:29:54.427670 24995 tablet_replica.cc:439] T e6bd5365020f400d8946803e5cdf5390 P 66291c8b70dc46a6acbfdf3c9c475500: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "66291c8b70dc46a6acbfdf3c9c475500" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a69b617c9f843b2a9001f6d598c63e8" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 38223 } } peers { permanent_uuid: "ffbd74368f6e483abebb4ff75b582ba0" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 41601 } } peers { permanent_uuid: "66291c8b70dc46a6acbfdf3c9c475500" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 38915 } } }
I20260926 07:29:54.427795 24995 tablet_replica.cc:442] T e6bd5365020f400d8946803e5cdf5390 P 66291c8b70dc46a6acbfdf3c9c475500: This TxnStatusTablet replica's current role is: LEADER
I20260926 07:29:54.428170 25010 mvcc.cc:205] Tried to move back new op lower bound from 7333510325919055872 to 7333510325763416064. Current Snapshot: MvccSnapshot[applied={T|T < 7333510325919055872}]
I20260926 07:29:54.429049 25012 mvcc.cc:205] Tried to move back new op lower bound from 7333510325919055872 to 7333510325763416064. Current Snapshot: MvccSnapshot[applied={T|T < 7333510325919055872}]
I20260926 07:29:54.429090 24991 tablet_replica.cc:439] T e6bd5365020f400d8946803e5cdf5390 P 3a69b617c9f843b2a9001f6d598c63e8: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "66291c8b70dc46a6acbfdf3c9c475500" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a69b617c9f843b2a9001f6d598c63e8" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 38223 } } peers { permanent_uuid: "ffbd74368f6e483abebb4ff75b582ba0" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 41601 } } peers { permanent_uuid: "66291c8b70dc46a6acbfdf3c9c475500" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 38915 } } }
I20260926 07:29:54.429216 24991 tablet_replica.cc:442] T e6bd5365020f400d8946803e5cdf5390 P 3a69b617c9f843b2a9001f6d598c63e8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:29:54.429221 25011 mvcc.cc:205] Tried to move back new op lower bound from 7333510325919055872 to 7333510325763416064. Current Snapshot: MvccSnapshot[applied={T|T < 7333510325919055872}]
I20260926 07:29:54.447037 24980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:54.442971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52924 (local address 127.22.53.195:38915)
0926 07:29:54.443123 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:54.443128 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:54.443273 (+ 145us) server_negotiation.cc:407] Connection header received
0926 07:29:54.443430 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:54.443434 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:54.443493 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:54.443580 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:29:54.443863 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:54.444773 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:54.445723 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:54.446077 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:54.446183 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:54.446450 (+ 267us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:29:54.446543 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:29:54.446619 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:54.446817 (+ 198us) server_negotiation.cc:299] Negotiation successful
0926 07:29:54.446877 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 07:29:54.461071 24980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:54.457503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.195:45581 (local address 127.22.53.195:38915)
0926 07:29:54.457644 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:54.457649 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:54.457813 (+ 164us) server_negotiation.cc:407] Connection header received
0926 07:29:54.457982 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:54.457984 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:54.458033 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:54.458107 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:29:54.458410 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:54.458924 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:54.459831 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:54.460046 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:54.460155 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:54.460480 (+ 325us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:29:54.460637 (+ 157us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:29:54.460737 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:54.460825 (+ 88us) server_negotiation.cc:299] Negotiation successful
0926 07:29:54.460880 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":66}
I20260926 07:29:54.461081 25022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:54.457352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.195:38915 (local address 127.22.53.195:45581)
0926 07:29:54.457726 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:54.457738 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 07:29:54.457874 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:29:54.458126 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:29:54.458128 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:29:54.458134 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 07:29:54.458302 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:54.458315 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:54.459031 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:54.459037 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:54.459708 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:54.459716 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:54.460161 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:54.460164 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:54.460263 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:54.460314 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 07:29:54.460741 (+ 427us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 07:29:54.460749 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 07:29:54.460816 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 07:29:54.460899 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"mutex_wait_us":20,"thread_start_us":108,"threads_started":1}
I20260926 07:29:54.469102 24776 raft_consensus.cc:1261] T 7340e0073345498da8a3a776bf0913fd P ffbd74368f6e483abebb4ff75b582ba0 [term 1 FOLLOWER]: Refusing update from remote peer 66291c8b70dc46a6acbfdf3c9c475500: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:29:54.469436 24636 raft_consensus.cc:1261] T 7340e0073345498da8a3a776bf0913fd P 3a69b617c9f843b2a9001f6d598c63e8 [term 1 FOLLOWER]: Refusing update from remote peer 66291c8b70dc46a6acbfdf3c9c475500: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:29:54.469871 24993 consensus_queue.cc:1035] T 7340e0073345498da8a3a776bf0913fd P 66291c8b70dc46a6acbfdf3c9c475500 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ffbd74368f6e483abebb4ff75b582ba0" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 41601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:29:54.470077 25023 consensus_queue.cc:1035] T 7340e0073345498da8a3a776bf0913fd P 66291c8b70dc46a6acbfdf3c9c475500 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a69b617c9f843b2a9001f6d598c63e8" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 38223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:29:54.520731 24478 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47056:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:29:54.521003 24478 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47056:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:29:54.523963 24478 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 96456e5ef902415d9f99e829c7a793a1: Sending DeleteTablet for 3 replicas of tablet 7340e0073345498da8a3a776bf0913fd
I20260926 07:29:54.525064 24756 tablet_service.cc:1514] Processing DeleteTablet for tablet 7340e0073345498da8a3a776bf0913fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:29:54 UTC) from {username='kudu', principal='kudu/127.22.53.254@KRBTEST.COM'} at 127.0.0.1:51260
I20260926 07:29:54.525112 24895 tablet_service.cc:1514] Processing DeleteTablet for tablet 7340e0073345498da8a3a776bf0913fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:29:54 UTC) from {username='kudu', principal='kudu/127.22.53.254@KRBTEST.COM'} at 127.0.0.1:52908
I20260926 07:29:54.525660 24616 tablet_service.cc:1514] Processing DeleteTablet for tablet 7340e0073345498da8a3a776bf0913fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:29:54 UTC) from {username='kudu', principal='kudu/127.22.53.254@KRBTEST.COM'} at 127.0.0.1:56258
I20260926 07:29:54.525723 25028 tablet_replica.cc:330] T 7340e0073345498da8a3a776bf0913fd P ffbd74368f6e483abebb4ff75b582ba0: stopping tablet replica
I20260926 07:29:54.526150 25028 raft_consensus.cc:2229] T 7340e0073345498da8a3a776bf0913fd P ffbd74368f6e483abebb4ff75b582ba0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:29:54.526196 25030 tablet_replica.cc:330] T 7340e0073345498da8a3a776bf0913fd P 3a69b617c9f843b2a9001f6d598c63e8: stopping tablet replica
I20260926 07:29:54.526450 25028 raft_consensus.cc:2258] T 7340e0073345498da8a3a776bf0913fd P ffbd74368f6e483abebb4ff75b582ba0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:29:54.526707 25030 raft_consensus.cc:2229] T 7340e0073345498da8a3a776bf0913fd P 3a69b617c9f843b2a9001f6d598c63e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:29:54.526893 25029 tablet_replica.cc:330] T 7340e0073345498da8a3a776bf0913fd P 66291c8b70dc46a6acbfdf3c9c475500: stopping tablet replica
I20260926 07:29:54.527041 25030 raft_consensus.cc:2258] T 7340e0073345498da8a3a776bf0913fd P 3a69b617c9f843b2a9001f6d598c63e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:29:54.527180 25029 raft_consensus.cc:2229] T 7340e0073345498da8a3a776bf0913fd P 66291c8b70dc46a6acbfdf3c9c475500 [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:29:54.527225 24478 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47056:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 07:29:54.527603 25029 raft_consensus.cc:2258] T 7340e0073345498da8a3a776bf0913fd P 66291c8b70dc46a6acbfdf3c9c475500 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260926 07:29:54.527750 24478 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:29:54.530197 25028 ts_tablet_manager.cc:1905] T 7340e0073345498da8a3a776bf0913fd P ffbd74368f6e483abebb4ff75b582ba0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:29:54.530210 25029 ts_tablet_manager.cc:1905] T 7340e0073345498da8a3a776bf0913fd P 66291c8b70dc46a6acbfdf3c9c475500: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:29:54.533890 25028 ts_tablet_manager.cc:1918] T 7340e0073345498da8a3a776bf0913fd P ffbd74368f6e483abebb4ff75b582ba0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 07:29:54.533927 25029 ts_tablet_manager.cc:1918] T 7340e0073345498da8a3a776bf0913fd P 66291c8b70dc46a6acbfdf3c9c475500: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 07:29:54.534050 25029 log.cc:1196] T 7340e0073345498da8a3a776bf0913fd P 66291c8b70dc46a6acbfdf3c9c475500: Deleting WAL directory at /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407783529717-22743-0/minicluster-data/ts-2/wal/wals/7340e0073345498da8a3a776bf0913fd
I20260926 07:29:54.534049 25028 log.cc:1196] T 7340e0073345498da8a3a776bf0913fd P ffbd74368f6e483abebb4ff75b582ba0: Deleting WAL directory at /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407783529717-22743-0/minicluster-data/ts-1/wal/wals/7340e0073345498da8a3a776bf0913fd
I20260926 07:29:54.534449 25028 ts_tablet_manager.cc:1939] T 7340e0073345498da8a3a776bf0913fd P ffbd74368f6e483abebb4ff75b582ba0: Deleting consensus metadata
I20260926 07:29:54.534781 24895 tablet_service.cc:1467] Processing CreateTablet for tablet 5900d8854fbc4c3aac93384e3e6aa99c (DEFAULT_TABLE table=test-table [id=6c7734b8bad24f9080720a2a642fd2fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:29:54.535099 24895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5900d8854fbc4c3aac93384e3e6aa99c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:29:54.535177 24616 tablet_service.cc:1467] Processing CreateTablet for tablet 5900d8854fbc4c3aac93384e3e6aa99c (DEFAULT_TABLE table=test-table [id=6c7734b8bad24f9080720a2a642fd2fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:29:54.535482 24616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5900d8854fbc4c3aac93384e3e6aa99c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:29:54.537072 24756 tablet_service.cc:1467] Processing CreateTablet for tablet 5900d8854fbc4c3aac93384e3e6aa99c (DEFAULT_TABLE table=test-table [id=6c7734b8bad24f9080720a2a642fd2fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:29:54.537328 24756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5900d8854fbc4c3aac93384e3e6aa99c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:29:54.537832 24987 tablet_bootstrap.cc:492] T 5900d8854fbc4c3aac93384e3e6aa99c P 66291c8b70dc46a6acbfdf3c9c475500: Bootstrap starting.
I20260926 07:29:54.538089 24463 catalog_manager.cc:4841] TS ffbd74368f6e483abebb4ff75b582ba0 (127.22.53.194:41601): tablet 7340e0073345498da8a3a776bf0913fd (table test-table [id=c951e9df9fc145479cd01464d14f1062]) successfully deleted
I20260926 07:29:54.538444 24985 tablet_bootstrap.cc:492] T 5900d8854fbc4c3aac93384e3e6aa99c P 3a69b617c9f843b2a9001f6d598c63e8: Bootstrap starting.
I20260926 07:29:54.538899 24987 tablet_bootstrap.cc:654] T 5900d8854fbc4c3aac93384e3e6aa99c P 66291c8b70dc46a6acbfdf3c9c475500: Neither blocks nor log segments found. Creating new log.
I20260926 07:29:54.539494 24985 tablet_bootstrap.cc:654] T 5900d8854fbc4c3aac93384e3e6aa99c P 3a69b617c9f843b2a9001f6d598c63e8: Neither blocks nor log segments found. Creating new log.
I20260926 07:29:54.540397 25029 ts_tablet_manager.cc:1939] T 7340e0073345498da8a3a776bf0913fd P 66291c8b70dc46a6acbfdf3c9c475500: Deleting consensus metadata
I20260926 07:29:54.541234 24985 tablet_bootstrap.cc:492] T 5900d8854fbc4c3aac93384e3e6aa99c P 3a69b617c9f843b2a9001f6d598c63e8: No bootstrap required, opened a new log
I20260926 07:29:54.541326 24985 ts_tablet_manager.cc:1397] T 5900d8854fbc4c3aac93384e3e6aa99c P 3a69b617c9f843b2a9001f6d598c63e8: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260926 07:29:54.541735 24463 catalog_manager.cc:4841] TS 66291c8b70dc46a6acbfdf3c9c475500 (127.22.53.195:38915): tablet 7340e0073345498da8a3a776bf0913fd (table test-table [id=c951e9df9fc145479cd01464d14f1062]) successfully deleted
I20260926 07:29:54.541798 24986 tablet_bootstrap.cc:492] T 5900d8854fbc4c3aac93384e3e6aa99c P ffbd74368f6e483abebb4ff75b582ba0: Bootstrap starting.
I20260926 07:29:54.541877 24985 raft_consensus.cc:348] T 5900d8854fbc4c3aac93384e3e6aa99c P 3a69b617c9f843b2a9001f6d598c63e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a69b617c9f843b2a9001f6d598c63e8" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 38223 } } peers { permanent_uuid: "ffbd74368f6e483abebb4ff75b582ba0" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 41601 } } peers { permanent_uuid: "66291c8b70dc46a6acbfdf3c9c475500" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 38915 } }
I20260926 07:29:54.542022 24985 raft_consensus.cc:374] T 5900d8854fbc4c3aac93384e3e6aa99c P 3a69b617c9f843b2a9001f6d598c63e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:29:54.542055 24985 raft_consensus.cc:729] T 5900d8854fbc4c3aac93384e3e6aa99c P 3a69b617c9f843b2a9001f6d598c63e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a69b617c9f843b2a9001f6d598c63e8, State: Initialized, Role: FOLLOWER
I20260926 07:29:54.542191 24985 consensus_queue.cc:260] T 5900d8854fbc4c3aac93384e3e6aa99c P 3a69b617c9f843b2a9001f6d598c63e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a69b617c9f843b2a9001f6d598c63e8" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 38223 } } peers { permanent_uuid: "ffbd74368f6e483abebb4ff75b582ba0" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 41601 } } peers { permanent_uuid: "66291c8b70dc46a6acbfdf3c9c475500" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 38915 } }
I20260926 07:29:54.542539 24985 ts_tablet_manager.cc:1428] T 5900d8854fbc4c3aac93384e3e6aa99c P 3a69b617c9f843b2a9001f6d598c63e8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260926 07:29:54.542865 24986 tablet_bootstrap.cc:654] T 5900d8854fbc4c3aac93384e3e6aa99c P ffbd74368f6e483abebb4ff75b582ba0: Neither blocks nor log segments found. Creating new log.
I20260926 07:29:54.544389 24986 tablet_bootstrap.cc:492] T 5900d8854fbc4c3aac93384e3e6aa99c P ffbd74368f6e483abebb4ff75b582ba0: No bootstrap required, opened a new log
I20260926 07:29:54.544497 24986 ts_tablet_manager.cc:1397] T 5900d8854fbc4c3aac93384e3e6aa99c P ffbd74368f6e483abebb4ff75b582ba0: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 07:29:54.545138 24986 raft_consensus.cc:348] T 5900d8854fbc4c3aac93384e3e6aa99c P ffbd74368f6e483abebb4ff75b582ba0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a69b617c9f843b2a9001f6d598c63e8" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 38223 } } peers { permanent_uuid: "ffbd74368f6e483abebb4ff75b582ba0" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 41601 } } peers { permanent_uuid: "66291c8b70dc46a6acbfdf3c9c475500" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 38915 } }
I20260926 07:29:54.545261 24986 raft_consensus.cc:374] T 5900d8854fbc4c3aac93384e3e6aa99c P ffbd74368f6e483abebb4ff75b582ba0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:29:54.545295 24986 raft_consensus.cc:729] T 5900d8854fbc4c3aac93384e3e6aa99c P ffbd74368f6e483abebb4ff75b582ba0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffbd74368f6e483abebb4ff75b582ba0, State: Initialized, Role: FOLLOWER
I20260926 07:29:54.545433 24987 tablet_bootstrap.cc:492] T 5900d8854fbc4c3aac93384e3e6aa99c P 66291c8b70dc46a6acbfdf3c9c475500: No bootstrap required, opened a new log
I20260926 07:29:54.545527 24987 ts_tablet_manager.cc:1397] T 5900d8854fbc4c3aac93384e3e6aa99c P 66291c8b70dc46a6acbfdf3c9c475500: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20260926 07:29:54.545601 24986 consensus_queue.cc:260] T 5900d8854fbc4c3aac93384e3e6aa99c P ffbd74368f6e483abebb4ff75b582ba0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a69b617c9f843b2a9001f6d598c63e8" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 38223 } } peers { permanent_uuid: "ffbd74368f6e483abebb4ff75b582ba0" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 41601 } } peers { permanent_uuid: "66291c8b70dc46a6acbfdf3c9c475500" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 38915 } }
I20260926 07:29:54.546020 24987 raft_consensus.cc:348] T 5900d8854fbc4c3aac93384e3e6aa99c P 66291c8b70dc46a6acbfdf3c9c475500 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a69b617c9f843b2a9001f6d598c63e8" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 38223 } } peers { permanent_uuid: "ffbd74368f6e483abebb4ff75b582ba0" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 41601 } } peers { permanent_uuid: "66291c8b70dc46a6acbfdf3c9c475500" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 38915 } }
I20260926 07:29:54.546142 24987 raft_consensus.cc:374] T 5900d8854fbc4c3aac93384e3e6aa99c P 66291c8b70dc46a6acbfdf3c9c475500 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:29:54.546176 24987 raft_consensus.cc:729] T 5900d8854fbc4c3aac93384e3e6aa99c P 66291c8b70dc46a6acbfdf3c9c475500 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66291c8b70dc46a6acbfdf3c9c475500, State: Initialized, Role: FOLLOWER
I20260926 07:29:54.546316 24987 consensus_queue.cc:260] T 5900d8854fbc4c3aac93384e3e6aa99c P 66291c8b70dc46a6acbfdf3c9c475500 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a69b617c9f843b2a9001f6d598c63e8" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 38223 } } peers { permanent_uuid: "ffbd74368f6e483abebb4ff75b582ba0" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 41601 } } peers { permanent_uuid: "66291c8b70dc46a6acbfdf3c9c475500" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 38915 } }
I20260926 07:29:54.546653 24986 ts_tablet_manager.cc:1428] T 5900d8854fbc4c3aac93384e3e6aa99c P ffbd74368f6e483abebb4ff75b582ba0: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260926 07:29:54.546653 24987 ts_tablet_manager.cc:1428] T 5900d8854fbc4c3aac93384e3e6aa99c P 66291c8b70dc46a6acbfdf3c9c475500: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 07:29:54.548264 25030 ts_tablet_manager.cc:1905] T 7340e0073345498da8a3a776bf0913fd P 3a69b617c9f843b2a9001f6d598c63e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:29:54.551671 25030 ts_tablet_manager.cc:1918] T 7340e0073345498da8a3a776bf0913fd P 3a69b617c9f843b2a9001f6d598c63e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 07:29:54.551820 25030 log.cc:1196] T 7340e0073345498da8a3a776bf0913fd P 3a69b617c9f843b2a9001f6d598c63e8: Deleting WAL directory at /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407783529717-22743-0/minicluster-data/ts-0/wal/wals/7340e0073345498da8a3a776bf0913fd
I20260926 07:29:54.552147 25030 ts_tablet_manager.cc:1939] T 7340e0073345498da8a3a776bf0913fd P 3a69b617c9f843b2a9001f6d598c63e8: Deleting consensus metadata
I20260926 07:29:54.553537 24465 catalog_manager.cc:4841] TS 3a69b617c9f843b2a9001f6d598c63e8 (127.22.53.193:38223): tablet 7340e0073345498da8a3a776bf0913fd (table test-table [id=c951e9df9fc145479cd01464d14f1062]) successfully deleted
I20260926 07:29:54.784464 24995 raft_consensus.cc:482] T 5900d8854fbc4c3aac93384e3e6aa99c P 66291c8b70dc46a6acbfdf3c9c475500 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:29:54.784657 24995 raft_consensus.cc:504] T 5900d8854fbc4c3aac93384e3e6aa99c P 66291c8b70dc46a6acbfdf3c9c475500 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a69b617c9f843b2a9001f6d598c63e8" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 38223 } } peers { permanent_uuid: "ffbd74368f6e483abebb4ff75b582ba0" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 41601 } } peers { permanent_uuid: "66291c8b70dc46a6acbfdf3c9c475500" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 38915 } }
I20260926 07:29:54.785085 24995 leader_election.cc:288] T 5900d8854fbc4c3aac93384e3e6aa99c P 66291c8b70dc46a6acbfdf3c9c475500 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3a69b617c9f843b2a9001f6d598c63e8 (127.22.53.193:38223), ffbd74368f6e483abebb4ff75b582ba0 (127.22.53.194:41601)
I20260926 07:29:54.785606 24636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5900d8854fbc4c3aac93384e3e6aa99c" candidate_uuid: "66291c8b70dc46a6acbfdf3c9c475500" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a69b617c9f843b2a9001f6d598c63e8" is_pre_election: true
I20260926 07:29:54.785630 24775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5900d8854fbc4c3aac93384e3e6aa99c" candidate_uuid: "66291c8b70dc46a6acbfdf3c9c475500" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffbd74368f6e483abebb4ff75b582ba0" is_pre_election: true
I20260926 07:29:54.785777 24636 raft_consensus.cc:2454] T 5900d8854fbc4c3aac93384e3e6aa99c P 3a69b617c9f843b2a9001f6d598c63e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66291c8b70dc46a6acbfdf3c9c475500 in term 0.
I20260926 07:29:54.785848 24775 raft_consensus.cc:2454] T 5900d8854fbc4c3aac93384e3e6aa99c P ffbd74368f6e483abebb4ff75b582ba0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66291c8b70dc46a6acbfdf3c9c475500 in term 0.
I20260926 07:29:54.786201 24847 leader_election.cc:302] T 5900d8854fbc4c3aac93384e3e6aa99c P 66291c8b70dc46a6acbfdf3c9c475500 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3a69b617c9f843b2a9001f6d598c63e8, 66291c8b70dc46a6acbfdf3c9c475500, ffbd74368f6e483abebb4ff75b582ba0; no voters:
I20260926 07:29:54.786442 24995 raft_consensus.cc:2781] T 5900d8854fbc4c3aac93384e3e6aa99c P 66291c8b70dc46a6acbfdf3c9c475500 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:29:54.786502 24995 raft_consensus.cc:482] T 5900d8854fbc4c3aac93384e3e6aa99c P 66291c8b70dc46a6acbfdf3c9c475500 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:29:54.786556 24995 raft_consensus.cc:3037] T 5900d8854fbc4c3aac93384e3e6aa99c P 66291c8b70dc46a6acbfdf3c9c475500 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:29:54.787732 24995 raft_consensus.cc:504] T 5900d8854fbc4c3aac93384e3e6aa99c P 66291c8b70dc46a6acbfdf3c9c475500 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a69b617c9f843b2a9001f6d598c63e8" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 38223 } } peers { permanent_uuid: "ffbd74368f6e483abebb4ff75b582ba0" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 41601 } } peers { permanent_uuid: "66291c8b70dc46a6acbfdf3c9c475500" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 38915 } }
I20260926 07:29:54.788143 24995 leader_election.cc:288] T 5900d8854fbc4c3aac93384e3e6aa99c P 66291c8b70dc46a6acbfdf3c9c475500 [CANDIDATE]: Term 1 election: Requested vote from peers 3a69b617c9f843b2a9001f6d598c63e8 (127.22.53.193:38223), ffbd74368f6e483abebb4ff75b582ba0 (127.22.53.194:41601)
I20260926 07:29:54.788640 24775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5900d8854fbc4c3aac93384e3e6aa99c" candidate_uuid: "66291c8b70dc46a6acbfdf3c9c475500" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffbd74368f6e483abebb4ff75b582ba0"
I20260926 07:29:54.788693 24636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5900d8854fbc4c3aac93384e3e6aa99c" candidate_uuid: "66291c8b70dc46a6acbfdf3c9c475500" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a69b617c9f843b2a9001f6d598c63e8"
I20260926 07:29:54.788810 24636 raft_consensus.cc:3037] T 5900d8854fbc4c3aac93384e3e6aa99c P 3a69b617c9f843b2a9001f6d598c63e8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:29:54.788807 24775 raft_consensus.cc:3037] T 5900d8854fbc4c3aac93384e3e6aa99c P ffbd74368f6e483abebb4ff75b582ba0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:29:54.789995 24636 raft_consensus.cc:2454] T 5900d8854fbc4c3aac93384e3e6aa99c P 3a69b617c9f843b2a9001f6d598c63e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66291c8b70dc46a6acbfdf3c9c475500 in term 1.
I20260926 07:29:54.789995 24775 raft_consensus.cc:2454] T 5900d8854fbc4c3aac93384e3e6aa99c P ffbd74368f6e483abebb4ff75b582ba0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66291c8b70dc46a6acbfdf3c9c475500 in term 1.
I20260926 07:29:54.790438 24849 leader_election.cc:302] T 5900d8854fbc4c3aac93384e3e6aa99c P 66291c8b70dc46a6acbfdf3c9c475500 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3a69b617c9f843b2a9001f6d598c63e8, 66291c8b70dc46a6acbfdf3c9c475500, ffbd74368f6e483abebb4ff75b582ba0; no voters:
I20260926 07:29:54.790704 24995 raft_consensus.cc:2781] T 5900d8854fbc4c3aac93384e3e6aa99c P 66291c8b70dc46a6acbfdf3c9c475500 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:29:54.790823 24995 raft_consensus.cc:686] T 5900d8854fbc4c3aac93384e3e6aa99c P 66291c8b70dc46a6acbfdf3c9c475500 [term 1 LEADER]: Becoming Leader. State: Replica: 66291c8b70dc46a6acbfdf3c9c475500, State: Running, Role: LEADER
I20260926 07:29:54.790999 24995 consensus_queue.cc:237] T 5900d8854fbc4c3aac93384e3e6aa99c P 66291c8b70dc46a6acbfdf3c9c475500 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a69b617c9f843b2a9001f6d598c63e8" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 38223 } } peers { permanent_uuid: "ffbd74368f6e483abebb4ff75b582ba0" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 41601 } } peers { permanent_uuid: "66291c8b70dc46a6acbfdf3c9c475500" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 38915 } }
I20260926 07:29:54.792862 24478 catalog_manager.cc:5495] T 5900d8854fbc4c3aac93384e3e6aa99c P 66291c8b70dc46a6acbfdf3c9c475500 reported cstate change: term changed from 0 to 1, leader changed from <none> to 66291c8b70dc46a6acbfdf3c9c475500 (127.22.53.195). New cstate: current_term: 1 leader_uuid: "66291c8b70dc46a6acbfdf3c9c475500" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a69b617c9f843b2a9001f6d598c63e8" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 38223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ffbd74368f6e483abebb4ff75b582ba0" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 41601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66291c8b70dc46a6acbfdf3c9c475500" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 38915 } health_report { overall_health: HEALTHY } } }
I20260926 07:29:54.861963 24775 raft_consensus.cc:1261] T 5900d8854fbc4c3aac93384e3e6aa99c P ffbd74368f6e483abebb4ff75b582ba0 [term 1 FOLLOWER]: Refusing update from remote peer 66291c8b70dc46a6acbfdf3c9c475500: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:29:54.862370 24636 raft_consensus.cc:1261] T 5900d8854fbc4c3aac93384e3e6aa99c P 3a69b617c9f843b2a9001f6d598c63e8 [term 1 FOLLOWER]: Refusing update from remote peer 66291c8b70dc46a6acbfdf3c9c475500: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:29:54.862404 24995 consensus_queue.cc:1035] T 5900d8854fbc4c3aac93384e3e6aa99c P 66291c8b70dc46a6acbfdf3c9c475500 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ffbd74368f6e483abebb4ff75b582ba0" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 41601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:29:54.862828 24995 consensus_queue.cc:1035] T 5900d8854fbc4c3aac93384e3e6aa99c P 66291c8b70dc46a6acbfdf3c9c475500 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a69b617c9f843b2a9001f6d598c63e8" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 38223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 07:29:54.877118 25010 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 07:29:56.444804 24479 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47056:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:29:56.444973 24479 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47056:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:29:56.447650 24479 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 96456e5ef902415d9f99e829c7a793a1: Sending DeleteTablet for 3 replicas of tablet 5900d8854fbc4c3aac93384e3e6aa99c
I20260926 07:29:56.448511 24616 tablet_service.cc:1514] Processing DeleteTablet for tablet 5900d8854fbc4c3aac93384e3e6aa99c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:29:56 UTC) from {username='kudu', principal='kudu/127.22.53.254@KRBTEST.COM'} at 127.0.0.1:56258
I20260926 07:29:56.448599 24895 tablet_service.cc:1514] Processing DeleteTablet for tablet 5900d8854fbc4c3aac93384e3e6aa99c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:29:56 UTC) from {username='kudu', principal='kudu/127.22.53.254@KRBTEST.COM'} at 127.0.0.1:52908
I20260926 07:29:56.448606 24756 tablet_service.cc:1514] Processing DeleteTablet for tablet 5900d8854fbc4c3aac93384e3e6aa99c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:29:56 UTC) from {username='kudu', principal='kudu/127.22.53.254@KRBTEST.COM'} at 127.0.0.1:51260
I20260926 07:29:56.448982 25059 tablet_replica.cc:330] T 5900d8854fbc4c3aac93384e3e6aa99c P ffbd74368f6e483abebb4ff75b582ba0: stopping tablet replica
I20260926 07:29:56.449124 25059 raft_consensus.cc:2229] T 5900d8854fbc4c3aac93384e3e6aa99c P ffbd74368f6e483abebb4ff75b582ba0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:29:56.449440 25059 raft_consensus.cc:2258] T 5900d8854fbc4c3aac93384e3e6aa99c P ffbd74368f6e483abebb4ff75b582ba0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:29:56.449744 25058 tablet_replica.cc:330] T 5900d8854fbc4c3aac93384e3e6aa99c P 3a69b617c9f843b2a9001f6d598c63e8: stopping tablet replica
I20260926 07:29:56.449975 25058 raft_consensus.cc:2229] T 5900d8854fbc4c3aac93384e3e6aa99c P 3a69b617c9f843b2a9001f6d598c63e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:29:56.450151 25058 raft_consensus.cc:2258] T 5900d8854fbc4c3aac93384e3e6aa99c P 3a69b617c9f843b2a9001f6d598c63e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:29:56.450256 25059 ts_tablet_manager.cc:1905] T 5900d8854fbc4c3aac93384e3e6aa99c P ffbd74368f6e483abebb4ff75b582ba0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:29:56.450960 25058 ts_tablet_manager.cc:1905] T 5900d8854fbc4c3aac93384e3e6aa99c P 3a69b617c9f843b2a9001f6d598c63e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:29:56.451071 25057 tablet_replica.cc:330] T 5900d8854fbc4c3aac93384e3e6aa99c P 66291c8b70dc46a6acbfdf3c9c475500: stopping tablet replica
I20260926 07:29:56.451257 25057 raft_consensus.cc:2229] T 5900d8854fbc4c3aac93384e3e6aa99c P 66291c8b70dc46a6acbfdf3c9c475500 [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:29:56.451545 25057 raft_consensus.cc:2258] T 5900d8854fbc4c3aac93384e3e6aa99c P 66291c8b70dc46a6acbfdf3c9c475500 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:29:56.452270 25057 ts_tablet_manager.cc:1905] T 5900d8854fbc4c3aac93384e3e6aa99c P 66291c8b70dc46a6acbfdf3c9c475500: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:29:56.453862 25058 ts_tablet_manager.cc:1918] T 5900d8854fbc4c3aac93384e3e6aa99c P 3a69b617c9f843b2a9001f6d598c63e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 07:29:56.453958 25058 log.cc:1196] T 5900d8854fbc4c3aac93384e3e6aa99c P 3a69b617c9f843b2a9001f6d598c63e8: Deleting WAL directory at /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407783529717-22743-0/minicluster-data/ts-0/wal/wals/5900d8854fbc4c3aac93384e3e6aa99c
I20260926 07:29:56.454324 25058 ts_tablet_manager.cc:1939] T 5900d8854fbc4c3aac93384e3e6aa99c P 3a69b617c9f843b2a9001f6d598c63e8: Deleting consensus metadata
I20260926 07:29:56.454491 25059 ts_tablet_manager.cc:1918] T 5900d8854fbc4c3aac93384e3e6aa99c P ffbd74368f6e483abebb4ff75b582ba0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 07:29:56.454597 25059 log.cc:1196] T 5900d8854fbc4c3aac93384e3e6aa99c P ffbd74368f6e483abebb4ff75b582ba0: Deleting WAL directory at /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407783529717-22743-0/minicluster-data/ts-1/wal/wals/5900d8854fbc4c3aac93384e3e6aa99c
I20260926 07:29:56.454881 25059 ts_tablet_manager.cc:1939] T 5900d8854fbc4c3aac93384e3e6aa99c P ffbd74368f6e483abebb4ff75b582ba0: Deleting consensus metadata
I20260926 07:29:56.455207 25057 ts_tablet_manager.cc:1918] T 5900d8854fbc4c3aac93384e3e6aa99c P 66291c8b70dc46a6acbfdf3c9c475500: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 07:29:56.455307 25057 log.cc:1196] T 5900d8854fbc4c3aac93384e3e6aa99c P 66291c8b70dc46a6acbfdf3c9c475500: Deleting WAL directory at /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407783529717-22743-0/minicluster-data/ts-2/wal/wals/5900d8854fbc4c3aac93384e3e6aa99c
I20260926 07:29:56.455477 24465 catalog_manager.cc:4841] TS 3a69b617c9f843b2a9001f6d598c63e8 (127.22.53.193:38223): tablet 5900d8854fbc4c3aac93384e3e6aa99c (table test-table [id=6c7734b8bad24f9080720a2a642fd2fc]) successfully deleted
I20260926 07:29:56.455621 25057 ts_tablet_manager.cc:1939] T 5900d8854fbc4c3aac93384e3e6aa99c P 66291c8b70dc46a6acbfdf3c9c475500: Deleting consensus metadata
I20260926 07:29:56.455735 24463 catalog_manager.cc:4841] TS ffbd74368f6e483abebb4ff75b582ba0 (127.22.53.194:41601): tablet 5900d8854fbc4c3aac93384e3e6aa99c (table test-table [id=6c7734b8bad24f9080720a2a642fd2fc]) successfully deleted
I20260926 07:29:56.456390 24463 catalog_manager.cc:4841] TS 66291c8b70dc46a6acbfdf3c9c475500 (127.22.53.195:38915): tablet 5900d8854fbc4c3aac93384e3e6aa99c (table test-table [id=6c7734b8bad24f9080720a2a642fd2fc]) successfully deleted
Sep 26 07:29:56 dist-test-slave-wdjc krb5kdc[24434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407796, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 07:29:56 dist-test-slave-wdjc krb5kdc[24434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407796, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.22.53.254@KRBTEST.COM
I20260926 07:29:56.479885 25067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:56.470616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47074 (local address 127.22.53.254:38529)
0926 07:29:56.470934 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:56.470941 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:56.470958 (+ 17us) server_negotiation.cc:407] Connection header received
0926 07:29:56.471088 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:56.471091 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:56.471149 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:56.471234 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:56.471965 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:56.472637 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:56.473503 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:56.473744 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:56.473850 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:56.476405 (+ 2555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:56.476408 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:56.476410 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:29:56.476441 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:56.478225 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:56.478682 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:56.478685 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:56.478686 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:56.478768 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:56.479026 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:56.479028 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:56.479029 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:56.479219 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:56.479335 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:56.479526 (+ 191us) server_negotiation.cc:299] Negotiation successful
0926 07:29:56.479638 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":115,"threads_started":1}
I20260926 07:29:56.482813 24479 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47074:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:29:56.483316 24479 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:29:56.488651 24616 tablet_service.cc:1467] Processing CreateTablet for tablet 1fc84875ef9a4376a1f459d4629df8ad (DEFAULT_TABLE table=test-table [id=74ab0bad4adc4f70bf2c364461f36beb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:29:56.488651 24756 tablet_service.cc:1467] Processing CreateTablet for tablet 1fc84875ef9a4376a1f459d4629df8ad (DEFAULT_TABLE table=test-table [id=74ab0bad4adc4f70bf2c364461f36beb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:29:56.488762 24895 tablet_service.cc:1467] Processing CreateTablet for tablet 1fc84875ef9a4376a1f459d4629df8ad (DEFAULT_TABLE table=test-table [id=74ab0bad4adc4f70bf2c364461f36beb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:29:56.488981 24616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fc84875ef9a4376a1f459d4629df8ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:29:56.488982 24756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fc84875ef9a4376a1f459d4629df8ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:29:56.489089 24895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fc84875ef9a4376a1f459d4629df8ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:29:56.491577 25068 tablet_bootstrap.cc:492] T 1fc84875ef9a4376a1f459d4629df8ad P 66291c8b70dc46a6acbfdf3c9c475500: Bootstrap starting.
I20260926 07:29:56.492172 25070 tablet_bootstrap.cc:492] T 1fc84875ef9a4376a1f459d4629df8ad P ffbd74368f6e483abebb4ff75b582ba0: Bootstrap starting.
I20260926 07:29:56.492715 25068 tablet_bootstrap.cc:654] T 1fc84875ef9a4376a1f459d4629df8ad P 66291c8b70dc46a6acbfdf3c9c475500: Neither blocks nor log segments found. Creating new log.
I20260926 07:29:56.493554 25070 tablet_bootstrap.cc:654] T 1fc84875ef9a4376a1f459d4629df8ad P ffbd74368f6e483abebb4ff75b582ba0: Neither blocks nor log segments found. Creating new log.
I20260926 07:29:56.494102 25068 tablet_bootstrap.cc:492] T 1fc84875ef9a4376a1f459d4629df8ad P 66291c8b70dc46a6acbfdf3c9c475500: No bootstrap required, opened a new log
I20260926 07:29:56.494207 25068 ts_tablet_manager.cc:1397] T 1fc84875ef9a4376a1f459d4629df8ad P 66291c8b70dc46a6acbfdf3c9c475500: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260926 07:29:56.494331 25069 tablet_bootstrap.cc:492] T 1fc84875ef9a4376a1f459d4629df8ad P 3a69b617c9f843b2a9001f6d598c63e8: Bootstrap starting.
I20260926 07:29:56.494764 25068 raft_consensus.cc:348] T 1fc84875ef9a4376a1f459d4629df8ad P 66291c8b70dc46a6acbfdf3c9c475500 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a69b617c9f843b2a9001f6d598c63e8" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 38223 } } peers { permanent_uuid: "ffbd74368f6e483abebb4ff75b582ba0" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 41601 } } peers { permanent_uuid: "66291c8b70dc46a6acbfdf3c9c475500" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 38915 } }
I20260926 07:29:56.494913 25068 raft_consensus.cc:374] T 1fc84875ef9a4376a1f459d4629df8ad P 66291c8b70dc46a6acbfdf3c9c475500 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:29:56.494961 25068 raft_consensus.cc:729] T 1fc84875ef9a4376a1f459d4629df8ad P 66291c8b70dc46a6acbfdf3c9c475500 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66291c8b70dc46a6acbfdf3c9c475500, State: Initialized, Role: FOLLOWER
I20260926 07:29:56.495102 25068 consensus_queue.cc:260] T 1fc84875ef9a4376a1f459d4629df8ad P 66291c8b70dc46a6acbfdf3c9c475500 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a69b617c9f843b2a9001f6d598c63e8" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 38223 } } peers { permanent_uuid: "ffbd74368f6e483abebb4ff75b582ba0" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 41601 } } peers { permanent_uuid: "66291c8b70dc46a6acbfdf3c9c475500" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 38915 } }
I20260926 07:29:56.495332 25069 tablet_bootstrap.cc:654] T 1fc84875ef9a4376a1f459d4629df8ad P 3a69b617c9f843b2a9001f6d598c63e8: Neither blocks nor log segments found. Creating new log.
I20260926 07:29:56.495517 25068 ts_tablet_manager.cc:1428] T 1fc84875ef9a4376a1f459d4629df8ad P 66291c8b70dc46a6acbfdf3c9c475500: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260926 07:29:56.496387 25070 tablet_bootstrap.cc:492] T 1fc84875ef9a4376a1f459d4629df8ad P ffbd74368f6e483abebb4ff75b582ba0: No bootstrap required, opened a new log
I20260926 07:29:56.496515 25070 ts_tablet_manager.cc:1397] T 1fc84875ef9a4376a1f459d4629df8ad P ffbd74368f6e483abebb4ff75b582ba0: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260926 07:29:56.496724 25069 tablet_bootstrap.cc:492] T 1fc84875ef9a4376a1f459d4629df8ad P 3a69b617c9f843b2a9001f6d598c63e8: No bootstrap required, opened a new log
I20260926 07:29:56.496837 25069 ts_tablet_manager.cc:1397] T 1fc84875ef9a4376a1f459d4629df8ad P 3a69b617c9f843b2a9001f6d598c63e8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 07:29:56.497078 25070 raft_consensus.cc:348] T 1fc84875ef9a4376a1f459d4629df8ad P ffbd74368f6e483abebb4ff75b582ba0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a69b617c9f843b2a9001f6d598c63e8" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 38223 } } peers { permanent_uuid: "ffbd74368f6e483abebb4ff75b582ba0" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 41601 } } peers { permanent_uuid: "66291c8b70dc46a6acbfdf3c9c475500" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 38915 } }
I20260926 07:29:56.497231 25070 raft_consensus.cc:374] T 1fc84875ef9a4376a1f459d4629df8ad P ffbd74368f6e483abebb4ff75b582ba0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:29:56.497270 25070 raft_consensus.cc:729] T 1fc84875ef9a4376a1f459d4629df8ad P ffbd74368f6e483abebb4ff75b582ba0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffbd74368f6e483abebb4ff75b582ba0, State: Initialized, Role: FOLLOWER
I20260926 07:29:56.497398 25070 consensus_queue.cc:260] T 1fc84875ef9a4376a1f459d4629df8ad P ffbd74368f6e483abebb4ff75b582ba0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a69b617c9f843b2a9001f6d598c63e8" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 38223 } } peers { permanent_uuid: "ffbd74368f6e483abebb4ff75b582ba0" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 41601 } } peers { permanent_uuid: "66291c8b70dc46a6acbfdf3c9c475500" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 38915 } }
I20260926 07:29:56.497817 25070 ts_tablet_manager.cc:1428] T 1fc84875ef9a4376a1f459d4629df8ad P ffbd74368f6e483abebb4ff75b582ba0: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260926 07:29:56.497789 25069 raft_consensus.cc:348] T 1fc84875ef9a4376a1f459d4629df8ad P 3a69b617c9f843b2a9001f6d598c63e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a69b617c9f843b2a9001f6d598c63e8" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 38223 } } peers { permanent_uuid: "ffbd74368f6e483abebb4ff75b582ba0" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 41601 } } peers { permanent_uuid: "66291c8b70dc46a6acbfdf3c9c475500" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 38915 } }
I20260926 07:29:56.497915 25069 raft_consensus.cc:374] T 1fc84875ef9a4376a1f459d4629df8ad P 3a69b617c9f843b2a9001f6d598c63e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:29:56.497948 25069 raft_consensus.cc:729] T 1fc84875ef9a4376a1f459d4629df8ad P 3a69b617c9f843b2a9001f6d598c63e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a69b617c9f843b2a9001f6d598c63e8, State: Initialized, Role: FOLLOWER
I20260926 07:29:56.498068 25069 consensus_queue.cc:260] T 1fc84875ef9a4376a1f459d4629df8ad P 3a69b617c9f843b2a9001f6d598c63e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a69b617c9f843b2a9001f6d598c63e8" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 38223 } } peers { permanent_uuid: "ffbd74368f6e483abebb4ff75b582ba0" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 41601 } } peers { permanent_uuid: "66291c8b70dc46a6acbfdf3c9c475500" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 38915 } }
I20260926 07:29:56.498454 25069 ts_tablet_manager.cc:1428] T 1fc84875ef9a4376a1f459d4629df8ad P 3a69b617c9f843b2a9001f6d598c63e8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 07:29:56.891247 25074 raft_consensus.cc:482] T 1fc84875ef9a4376a1f459d4629df8ad P ffbd74368f6e483abebb4ff75b582ba0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:29:56.891412 25074 raft_consensus.cc:504] T 1fc84875ef9a4376a1f459d4629df8ad P ffbd74368f6e483abebb4ff75b582ba0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a69b617c9f843b2a9001f6d598c63e8" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 38223 } } peers { permanent_uuid: "ffbd74368f6e483abebb4ff75b582ba0" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 41601 } } peers { permanent_uuid: "66291c8b70dc46a6acbfdf3c9c475500" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 38915 } }
I20260926 07:29:56.892761 25074 leader_election.cc:288] T 1fc84875ef9a4376a1f459d4629df8ad P ffbd74368f6e483abebb4ff75b582ba0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3a69b617c9f843b2a9001f6d598c63e8 (127.22.53.193:38223), 66291c8b70dc46a6acbfdf3c9c475500 (127.22.53.195:38915)
I20260926 07:29:56.897832 25077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:56.893204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.194:49393 (local address 127.22.53.195:38915)
0926 07:29:56.893550 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:56.893558 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:56.893822 (+ 264us) server_negotiation.cc:407] Connection header received
0926 07:29:56.893879 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:56.893881 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:56.893937 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:56.894024 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:29:56.894337 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:56.894965 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:56.896498 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:56.897153 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:56.897303 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:56.897363 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:56.897576 (+ 213us) server_negotiation.cc:299] Negotiation successful
0926 07:29:56.897676 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":118,"threads_started":1}
I20260926 07:29:56.897855 25076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:56.893123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.195:38915 (local address 127.22.53.194:49393)
0926 07:29:56.893583 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:56.893599 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:29:56.893724 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:29:56.894056 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:29:56.894059 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:29:56.894074 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:29:56.894214 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:56.894219 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:56.895093 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:56.895098 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:56.896362 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:56.896373 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:56.897315 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:56.897319 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:56.897423 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:56.897452 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 07:29:56.897556 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 07:29:56.897687 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"mutex_wait_us":30,"thread_start_us":146,"threads_started":1}
I20260926 07:29:56.897925 25078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:56.893204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.193:38223 (local address 127.22.53.194:57417)
0926 07:29:56.893693 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:56.893706 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:29:56.893799 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:29:56.894173 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:29:56.894176 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:29:56.894196 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:29:56.894355 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:56.894363 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:56.895527 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:56.895531 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:56.896646 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:56.896653 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:56.897469 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:56.897472 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:56.897575 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:56.897619 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 07:29:56.897696 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 07:29:56.897771 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"mutex_wait_us":1,"thread_start_us":124,"threads_started":1}
I20260926 07:29:56.898325 25079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:56.893560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.194:57417 (local address 127.22.53.193:38223)
0926 07:29:56.893912 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:56.893922 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:56.893936 (+ 14us) server_negotiation.cc:407] Connection header received
0926 07:29:56.893987 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:56.893990 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:56.894041 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:56.894117 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:29:56.894526 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:56.895378 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:56.896763 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:56.897203 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:56.897357 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:56.897419 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:56.898085 (+ 666us) server_negotiation.cc:299] Negotiation successful
0926 07:29:56.898176 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":130,"threads_started":1}
I20260926 07:29:56.898773 24635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fc84875ef9a4376a1f459d4629df8ad" candidate_uuid: "ffbd74368f6e483abebb4ff75b582ba0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a69b617c9f843b2a9001f6d598c63e8" is_pre_election: true
I20260926 07:29:56.898887 24916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fc84875ef9a4376a1f459d4629df8ad" candidate_uuid: "ffbd74368f6e483abebb4ff75b582ba0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66291c8b70dc46a6acbfdf3c9c475500" is_pre_election: true
I20260926 07:29:56.898973 24635 raft_consensus.cc:2454] T 1fc84875ef9a4376a1f459d4629df8ad P 3a69b617c9f843b2a9001f6d598c63e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ffbd74368f6e483abebb4ff75b582ba0 in term 0.
I20260926 07:29:56.899127 24916 raft_consensus.cc:2454] T 1fc84875ef9a4376a1f459d4629df8ad P 66291c8b70dc46a6acbfdf3c9c475500 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ffbd74368f6e483abebb4ff75b582ba0 in term 0.
I20260926 07:29:56.899415 24709 leader_election.cc:302] T 1fc84875ef9a4376a1f459d4629df8ad P ffbd74368f6e483abebb4ff75b582ba0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a69b617c9f843b2a9001f6d598c63e8, ffbd74368f6e483abebb4ff75b582ba0; no voters:
I20260926 07:29:56.899705 25074 raft_consensus.cc:2781] T 1fc84875ef9a4376a1f459d4629df8ad P ffbd74368f6e483abebb4ff75b582ba0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:29:56.899766 25074 raft_consensus.cc:482] T 1fc84875ef9a4376a1f459d4629df8ad P ffbd74368f6e483abebb4ff75b582ba0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:29:56.899804 25074 raft_consensus.cc:3037] T 1fc84875ef9a4376a1f459d4629df8ad P ffbd74368f6e483abebb4ff75b582ba0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:29:56.900976 25074 raft_consensus.cc:504] T 1fc84875ef9a4376a1f459d4629df8ad P ffbd74368f6e483abebb4ff75b582ba0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a69b617c9f843b2a9001f6d598c63e8" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 38223 } } peers { permanent_uuid: "ffbd74368f6e483abebb4ff75b582ba0" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 41601 } } peers { permanent_uuid: "66291c8b70dc46a6acbfdf3c9c475500" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 38915 } }
I20260926 07:29:56.901440 25074 leader_election.cc:288] T 1fc84875ef9a4376a1f459d4629df8ad P ffbd74368f6e483abebb4ff75b582ba0 [CANDIDATE]: Term 1 election: Requested vote from peers 3a69b617c9f843b2a9001f6d598c63e8 (127.22.53.193:38223), 66291c8b70dc46a6acbfdf3c9c475500 (127.22.53.195:38915)
I20260926 07:29:56.901842 24635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fc84875ef9a4376a1f459d4629df8ad" candidate_uuid: "ffbd74368f6e483abebb4ff75b582ba0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a69b617c9f843b2a9001f6d598c63e8"
I20260926 07:29:56.901984 24635 raft_consensus.cc:3037] T 1fc84875ef9a4376a1f459d4629df8ad P 3a69b617c9f843b2a9001f6d598c63e8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:29:56.903257 24635 raft_consensus.cc:2454] T 1fc84875ef9a4376a1f459d4629df8ad P 3a69b617c9f843b2a9001f6d598c63e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ffbd74368f6e483abebb4ff75b582ba0 in term 1.
I20260926 07:29:56.903553 24916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fc84875ef9a4376a1f459d4629df8ad" candidate_uuid: "ffbd74368f6e483abebb4ff75b582ba0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66291c8b70dc46a6acbfdf3c9c475500"
I20260926 07:29:56.903605 24709 leader_election.cc:302] T 1fc84875ef9a4376a1f459d4629df8ad P ffbd74368f6e483abebb4ff75b582ba0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a69b617c9f843b2a9001f6d598c63e8, ffbd74368f6e483abebb4ff75b582ba0; no voters:
I20260926 07:29:56.903680 24916 raft_consensus.cc:3037] T 1fc84875ef9a4376a1f459d4629df8ad P 66291c8b70dc46a6acbfdf3c9c475500 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:29:56.903825 25074 raft_consensus.cc:2781] T 1fc84875ef9a4376a1f459d4629df8ad P ffbd74368f6e483abebb4ff75b582ba0 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:29:56.904117 25074 raft_consensus.cc:686] T 1fc84875ef9a4376a1f459d4629df8ad P ffbd74368f6e483abebb4ff75b582ba0 [term 1 LEADER]: Becoming Leader. State: Replica: ffbd74368f6e483abebb4ff75b582ba0, State: Running, Role: LEADER
I20260926 07:29:56.904409 25074 consensus_queue.cc:237] T 1fc84875ef9a4376a1f459d4629df8ad P ffbd74368f6e483abebb4ff75b582ba0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a69b617c9f843b2a9001f6d598c63e8" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 38223 } } peers { permanent_uuid: "ffbd74368f6e483abebb4ff75b582ba0" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 41601 } } peers { permanent_uuid: "66291c8b70dc46a6acbfdf3c9c475500" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 38915 } }
I20260926 07:29:56.904917 24916 raft_consensus.cc:2454] T 1fc84875ef9a4376a1f459d4629df8ad P 66291c8b70dc46a6acbfdf3c9c475500 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ffbd74368f6e483abebb4ff75b582ba0 in term 1.
I20260926 07:29:56.907474 24474 catalog_manager.cc:5495] T 1fc84875ef9a4376a1f459d4629df8ad P ffbd74368f6e483abebb4ff75b582ba0 reported cstate change: term changed from 0 to 1, leader changed from <none> to ffbd74368f6e483abebb4ff75b582ba0 (127.22.53.194). New cstate: current_term: 1 leader_uuid: "ffbd74368f6e483abebb4ff75b582ba0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a69b617c9f843b2a9001f6d598c63e8" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 38223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ffbd74368f6e483abebb4ff75b582ba0" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 41601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66291c8b70dc46a6acbfdf3c9c475500" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 38915 } health_report { overall_health: UNKNOWN } } }
I20260926 07:29:56.962420 25091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:56.958124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51266 (local address 127.22.53.194:41601)
0926 07:29:56.958341 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:56.958347 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:56.958364 (+ 17us) server_negotiation.cc:407] Connection header received
0926 07:29:56.958471 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:56.958475 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:56.958546 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:56.958679 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:29:56.959015 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:56.959582 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:56.960489 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:56.960885 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:56.961074 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:56.961423 (+ 349us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:29:56.961926 (+ 503us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:29:56.962067 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:56.962158 (+ 91us) server_negotiation.cc:299] Negotiation successful
0926 07:29:56.962234 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":67,"threads_started":1}
I20260926 07:29:56.973896 25077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:56.970354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.194:60477 (local address 127.22.53.195:38915)
0926 07:29:56.970510 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:56.970514 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:56.970679 (+ 165us) server_negotiation.cc:407] Connection header received
0926 07:29:56.970862 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:56.970869 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:56.970915 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:56.970996 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:29:56.971268 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:56.971801 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:56.972770 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:56.972991 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:56.973105 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:56.973419 (+ 314us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:29:56.973498 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:29:56.973577 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:56.973672 (+ 95us) server_negotiation.cc:299] Negotiation successful
0926 07:29:56.973732 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 07:29:56.973917 25094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:56.970214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.195:38915 (local address 127.22.53.194:60477)
0926 07:29:56.970581 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:56.970604 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 07:29:56.970740 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:29:56.971019 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:29:56.971021 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:29:56.971029 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 07:29:56.971163 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:56.971168 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:56.971915 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:56.971922 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:56.972653 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:56.972662 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:56.973109 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:56.973113 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:56.973209 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:56.973269 (+ 60us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 07:29:56.973593 (+ 324us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 07:29:56.973600 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 07:29:56.973661 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 07:29:56.973745 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":119,"threads_started":1}
I20260926 07:29:56.981150 24635 raft_consensus.cc:1261] T 1fc84875ef9a4376a1f459d4629df8ad P 3a69b617c9f843b2a9001f6d598c63e8 [term 1 FOLLOWER]: Refusing update from remote peer ffbd74368f6e483abebb4ff75b582ba0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:29:56.981531 24916 raft_consensus.cc:1261] T 1fc84875ef9a4376a1f459d4629df8ad P 66291c8b70dc46a6acbfdf3c9c475500 [term 1 FOLLOWER]: Refusing update from remote peer ffbd74368f6e483abebb4ff75b582ba0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:29:56.981755 25080 consensus_queue.cc:1035] T 1fc84875ef9a4376a1f459d4629df8ad P ffbd74368f6e483abebb4ff75b582ba0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a69b617c9f843b2a9001f6d598c63e8" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 38223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:29:56.982074 25074 consensus_queue.cc:1035] T 1fc84875ef9a4376a1f459d4629df8ad P ffbd74368f6e483abebb4ff75b582ba0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66291c8b70dc46a6acbfdf3c9c475500" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 38915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:29:57.001513 25091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:56.997730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.195:34199 (local address 127.22.53.194:41601)
0926 07:29:56.997852 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:56.997856 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:56.998199 (+ 343us) server_negotiation.cc:407] Connection header received
0926 07:29:56.998353 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:56.998356 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:56.998408 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:56.998482 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:29:56.998767 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:56.999291 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:57.000337 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:57.000685 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:57.000801 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:57.001066 (+ 265us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:29:57.001165 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:29:57.001244 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:57.001342 (+ 98us) server_negotiation.cc:299] Negotiation successful
0926 07:29:57.001398 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 07:29:57.001530 25099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:56.997647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.194:41601 (local address 127.22.53.195:34199)
0926 07:29:56.998107 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:56.998123 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:29:56.998243 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:29:56.998502 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:29:56.998504 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:29:56.998512 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 07:29:56.998624 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:56.998631 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:56.999410 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:56.999415 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:57.000229 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:57.000243 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:57.000797 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:57.000801 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:57.000890 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:57.000926 (+ 36us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 07:29:57.001263 (+ 337us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 07:29:57.001269 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 07:29:57.001326 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 07:29:57.001411 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":86,"threads_started":1}
I20260926 07:29:57.025540 24478 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47074:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:29:57.025740 24478 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47074:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:29:57.028144 24478 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 96456e5ef902415d9f99e829c7a793a1: Sending DeleteTablet for 3 replicas of tablet 1fc84875ef9a4376a1f459d4629df8ad
I20260926 07:29:57.028898 24616 tablet_service.cc:1514] Processing DeleteTablet for tablet 1fc84875ef9a4376a1f459d4629df8ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:29:57 UTC) from {username='kudu', principal='kudu/127.22.53.254@KRBTEST.COM'} at 127.0.0.1:56258
I20260926 07:29:57.028959 24895 tablet_service.cc:1514] Processing DeleteTablet for tablet 1fc84875ef9a4376a1f459d4629df8ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:29:57 UTC) from {username='kudu', principal='kudu/127.22.53.254@KRBTEST.COM'} at 127.0.0.1:52908
I20260926 07:29:57.029318 25102 tablet_replica.cc:330] T 1fc84875ef9a4376a1f459d4629df8ad P 3a69b617c9f843b2a9001f6d598c63e8: stopping tablet replica
I20260926 07:29:57.029472 25102 raft_consensus.cc:2229] T 1fc84875ef9a4376a1f459d4629df8ad P 3a69b617c9f843b2a9001f6d598c63e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:29:57.029611 25102 raft_consensus.cc:2258] T 1fc84875ef9a4376a1f459d4629df8ad P 3a69b617c9f843b2a9001f6d598c63e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:29:57.029968 24756 tablet_service.cc:1514] Processing DeleteTablet for tablet 1fc84875ef9a4376a1f459d4629df8ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:29:57 UTC) from {username='kudu', principal='kudu/127.22.53.254@KRBTEST.COM'} at 127.0.0.1:51260
I20260926 07:29:57.030298 25104 tablet_replica.cc:330] T 1fc84875ef9a4376a1f459d4629df8ad P ffbd74368f6e483abebb4ff75b582ba0: stopping tablet replica
I20260926 07:29:57.030344 25102 ts_tablet_manager.cc:1905] T 1fc84875ef9a4376a1f459d4629df8ad P 3a69b617c9f843b2a9001f6d598c63e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:29:57.030385 24478 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47074:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 07:29:57.030511 25104 raft_consensus.cc:2229] T 1fc84875ef9a4376a1f459d4629df8ad P ffbd74368f6e483abebb4ff75b582ba0 [term 1 LEADER]: Raft consensus shutting down.
W20260926 07:29:57.030876 24478 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:29:57.031230 25104 raft_consensus.cc:2258] T 1fc84875ef9a4376a1f459d4629df8ad P ffbd74368f6e483abebb4ff75b582ba0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:29:57.031805 25103 tablet_replica.cc:330] T 1fc84875ef9a4376a1f459d4629df8ad P 66291c8b70dc46a6acbfdf3c9c475500: stopping tablet replica
I20260926 07:29:57.031952 25103 raft_consensus.cc:2229] T 1fc84875ef9a4376a1f459d4629df8ad P 66291c8b70dc46a6acbfdf3c9c475500 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:29:57.032096 25103 raft_consensus.cc:2258] T 1fc84875ef9a4376a1f459d4629df8ad P 66291c8b70dc46a6acbfdf3c9c475500 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:29:57.032433 25104 ts_tablet_manager.cc:1905] T 1fc84875ef9a4376a1f459d4629df8ad P ffbd74368f6e483abebb4ff75b582ba0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:29:57.034125 25102 ts_tablet_manager.cc:1918] T 1fc84875ef9a4376a1f459d4629df8ad P 3a69b617c9f843b2a9001f6d598c63e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 07:29:57.034224 25102 log.cc:1196] T 1fc84875ef9a4376a1f459d4629df8ad P 3a69b617c9f843b2a9001f6d598c63e8: Deleting WAL directory at /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407783529717-22743-0/minicluster-data/ts-0/wal/wals/1fc84875ef9a4376a1f459d4629df8ad
I20260926 07:29:57.034544 25102 ts_tablet_manager.cc:1939] T 1fc84875ef9a4376a1f459d4629df8ad P 3a69b617c9f843b2a9001f6d598c63e8: Deleting consensus metadata
I20260926 07:29:57.035140 25103 ts_tablet_manager.cc:1905] T 1fc84875ef9a4376a1f459d4629df8ad P 66291c8b70dc46a6acbfdf3c9c475500: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:29:57.035676 25104 ts_tablet_manager.cc:1918] T 1fc84875ef9a4376a1f459d4629df8ad P ffbd74368f6e483abebb4ff75b582ba0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 07:29:57.035806 25104 log.cc:1196] T 1fc84875ef9a4376a1f459d4629df8ad P ffbd74368f6e483abebb4ff75b582ba0: Deleting WAL directory at /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407783529717-22743-0/minicluster-data/ts-1/wal/wals/1fc84875ef9a4376a1f459d4629df8ad
I20260926 07:29:57.036125 25104 ts_tablet_manager.cc:1939] T 1fc84875ef9a4376a1f459d4629df8ad P ffbd74368f6e483abebb4ff75b582ba0: Deleting consensus metadata
I20260926 07:29:57.037066 24463 catalog_manager.cc:4841] TS ffbd74368f6e483abebb4ff75b582ba0 (127.22.53.194:41601): tablet 1fc84875ef9a4376a1f459d4629df8ad (table test-table [id=74ab0bad4adc4f70bf2c364461f36beb]) successfully deleted
I20260926 07:29:57.038090 25103 ts_tablet_manager.cc:1918] T 1fc84875ef9a4376a1f459d4629df8ad P 66291c8b70dc46a6acbfdf3c9c475500: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 07:29:57.038178 25103 log.cc:1196] T 1fc84875ef9a4376a1f459d4629df8ad P 66291c8b70dc46a6acbfdf3c9c475500: Deleting WAL directory at /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407783529717-22743-0/minicluster-data/ts-2/wal/wals/1fc84875ef9a4376a1f459d4629df8ad
I20260926 07:29:57.038460 25103 ts_tablet_manager.cc:1939] T 1fc84875ef9a4376a1f459d4629df8ad P 66291c8b70dc46a6acbfdf3c9c475500: Deleting consensus metadata
I20260926 07:29:57.038506 24616 tablet_service.cc:1467] Processing CreateTablet for tablet a309a1f586c445589592c408a3e32f3e (DEFAULT_TABLE table=test-table [id=2880847e76f14f228cdf31876d5d523e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:29:57.038690 24756 tablet_service.cc:1467] Processing CreateTablet for tablet a309a1f586c445589592c408a3e32f3e (DEFAULT_TABLE table=test-table [id=2880847e76f14f228cdf31876d5d523e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:29:57.038786 24616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a309a1f586c445589592c408a3e32f3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:29:57.038954 24756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a309a1f586c445589592c408a3e32f3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:29:57.039212 24463 catalog_manager.cc:4841] TS 66291c8b70dc46a6acbfdf3c9c475500 (127.22.53.195:38915): tablet 1fc84875ef9a4376a1f459d4629df8ad (table test-table [id=74ab0bad4adc4f70bf2c364461f36beb]) successfully deleted
I20260926 07:29:57.040220 24895 tablet_service.cc:1467] Processing CreateTablet for tablet a309a1f586c445589592c408a3e32f3e (DEFAULT_TABLE table=test-table [id=2880847e76f14f228cdf31876d5d523e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:29:57.040479 24895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a309a1f586c445589592c408a3e32f3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:29:57.040633 24465 catalog_manager.cc:4841] TS 3a69b617c9f843b2a9001f6d598c63e8 (127.22.53.193:38223): tablet 1fc84875ef9a4376a1f459d4629df8ad (table test-table [id=74ab0bad4adc4f70bf2c364461f36beb]) successfully deleted
I20260926 07:29:57.042001 25106 tablet_bootstrap.cc:492] T a309a1f586c445589592c408a3e32f3e P 3a69b617c9f843b2a9001f6d598c63e8: Bootstrap starting.
I20260926 07:29:57.043104 25106 tablet_bootstrap.cc:654] T a309a1f586c445589592c408a3e32f3e P 3a69b617c9f843b2a9001f6d598c63e8: Neither blocks nor log segments found. Creating new log.
I20260926 07:29:57.043706 25105 tablet_bootstrap.cc:492] T a309a1f586c445589592c408a3e32f3e P ffbd74368f6e483abebb4ff75b582ba0: Bootstrap starting.
I20260926 07:29:57.044739 25107 tablet_bootstrap.cc:492] T a309a1f586c445589592c408a3e32f3e P 66291c8b70dc46a6acbfdf3c9c475500: Bootstrap starting.
I20260926 07:29:57.044857 25105 tablet_bootstrap.cc:654] T a309a1f586c445589592c408a3e32f3e P ffbd74368f6e483abebb4ff75b582ba0: Neither blocks nor log segments found. Creating new log.
I20260926 07:29:57.045794 25107 tablet_bootstrap.cc:654] T a309a1f586c445589592c408a3e32f3e P 66291c8b70dc46a6acbfdf3c9c475500: Neither blocks nor log segments found. Creating new log.
I20260926 07:29:57.046219 25106 tablet_bootstrap.cc:492] T a309a1f586c445589592c408a3e32f3e P 3a69b617c9f843b2a9001f6d598c63e8: No bootstrap required, opened a new log
I20260926 07:29:57.046334 25106 ts_tablet_manager.cc:1397] T a309a1f586c445589592c408a3e32f3e P 3a69b617c9f843b2a9001f6d598c63e8: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260926 07:29:57.046887 25106 raft_consensus.cc:348] T a309a1f586c445589592c408a3e32f3e P 3a69b617c9f843b2a9001f6d598c63e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a69b617c9f843b2a9001f6d598c63e8" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 38223 } } peers { permanent_uuid: "ffbd74368f6e483abebb4ff75b582ba0" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 41601 } } peers { permanent_uuid: "66291c8b70dc46a6acbfdf3c9c475500" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 38915 } }
I20260926 07:29:57.047008 25105 tablet_bootstrap.cc:492] T a309a1f586c445589592c408a3e32f3e P ffbd74368f6e483abebb4ff75b582ba0: No bootstrap required, opened a new log
I20260926 07:29:57.047025 25106 raft_consensus.cc:374] T a309a1f586c445589592c408a3e32f3e P 3a69b617c9f843b2a9001f6d598c63e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:29:57.047058 25106 raft_consensus.cc:729] T a309a1f586c445589592c408a3e32f3e P 3a69b617c9f843b2a9001f6d598c63e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a69b617c9f843b2a9001f6d598c63e8, State: Initialized, Role: FOLLOWER
I20260926 07:29:57.047088 25105 ts_tablet_manager.cc:1397] T a309a1f586c445589592c408a3e32f3e P ffbd74368f6e483abebb4ff75b582ba0: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 07:29:57.047190 25106 consensus_queue.cc:260] T a309a1f586c445589592c408a3e32f3e P 3a69b617c9f843b2a9001f6d598c63e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a69b617c9f843b2a9001f6d598c63e8" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 38223 } } peers { permanent_uuid: "ffbd74368f6e483abebb4ff75b582ba0" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 41601 } } peers { permanent_uuid: "66291c8b70dc46a6acbfdf3c9c475500" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 38915 } }
I20260926 07:29:57.047502 25106 ts_tablet_manager.cc:1428] T a309a1f586c445589592c408a3e32f3e P 3a69b617c9f843b2a9001f6d598c63e8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260926 07:29:57.048036 25107 tablet_bootstrap.cc:492] T a309a1f586c445589592c408a3e32f3e P 66291c8b70dc46a6acbfdf3c9c475500: No bootstrap required, opened a new log
I20260926 07:29:57.048173 25107 ts_tablet_manager.cc:1397] T a309a1f586c445589592c408a3e32f3e P 66291c8b70dc46a6acbfdf3c9c475500: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 07:29:57.047549 25105 raft_consensus.cc:348] T a309a1f586c445589592c408a3e32f3e P ffbd74368f6e483abebb4ff75b582ba0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a69b617c9f843b2a9001f6d598c63e8" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 38223 } } peers { permanent_uuid: "ffbd74368f6e483abebb4ff75b582ba0" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 41601 } } peers { permanent_uuid: "66291c8b70dc46a6acbfdf3c9c475500" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 38915 } }
I20260926 07:29:57.048444 25105 raft_consensus.cc:374] T a309a1f586c445589592c408a3e32f3e P ffbd74368f6e483abebb4ff75b582ba0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:29:57.048553 25105 raft_consensus.cc:729] T a309a1f586c445589592c408a3e32f3e P ffbd74368f6e483abebb4ff75b582ba0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffbd74368f6e483abebb4ff75b582ba0, State: Initialized, Role: FOLLOWER
I20260926 07:29:57.048720 25107 raft_consensus.cc:348] T a309a1f586c445589592c408a3e32f3e P 66291c8b70dc46a6acbfdf3c9c475500 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a69b617c9f843b2a9001f6d598c63e8" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 38223 } } peers { permanent_uuid: "ffbd74368f6e483abebb4ff75b582ba0" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 41601 } } peers { permanent_uuid: "66291c8b70dc46a6acbfdf3c9c475500" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 38915 } }
I20260926 07:29:57.048846 25107 raft_consensus.cc:374] T a309a1f586c445589592c408a3e32f3e P 66291c8b70dc46a6acbfdf3c9c475500 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:29:57.048882 25107 raft_consensus.cc:729] T a309a1f586c445589592c408a3e32f3e P 66291c8b70dc46a6acbfdf3c9c475500 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66291c8b70dc46a6acbfdf3c9c475500, State: Initialized, Role: FOLLOWER
I20260926 07:29:57.048799 25105 consensus_queue.cc:260] T a309a1f586c445589592c408a3e32f3e P ffbd74368f6e483abebb4ff75b582ba0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a69b617c9f843b2a9001f6d598c63e8" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 38223 } } peers { permanent_uuid: "ffbd74368f6e483abebb4ff75b582ba0" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 41601 } } peers { permanent_uuid: "66291c8b70dc46a6acbfdf3c9c475500" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 38915 } }
I20260926 07:29:57.049012 25107 consensus_queue.cc:260] T a309a1f586c445589592c408a3e32f3e P 66291c8b70dc46a6acbfdf3c9c475500 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a69b617c9f843b2a9001f6d598c63e8" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 38223 } } peers { permanent_uuid: "ffbd74368f6e483abebb4ff75b582ba0" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 41601 } } peers { permanent_uuid: "66291c8b70dc46a6acbfdf3c9c475500" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 38915 } }
I20260926 07:29:57.049304 25107 ts_tablet_manager.cc:1428] T a309a1f586c445589592c408a3e32f3e P 66291c8b70dc46a6acbfdf3c9c475500: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 07:29:57.049329 25105 ts_tablet_manager.cc:1428] T a309a1f586c445589592c408a3e32f3e P ffbd74368f6e483abebb4ff75b582ba0: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260926 07:29:57.120467 25083 raft_consensus.cc:482] T a309a1f586c445589592c408a3e32f3e P 66291c8b70dc46a6acbfdf3c9c475500 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:29:57.120657 25083 raft_consensus.cc:504] T a309a1f586c445589592c408a3e32f3e P 66291c8b70dc46a6acbfdf3c9c475500 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a69b617c9f843b2a9001f6d598c63e8" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 38223 } } peers { permanent_uuid: "ffbd74368f6e483abebb4ff75b582ba0" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 41601 } } peers { permanent_uuid: "66291c8b70dc46a6acbfdf3c9c475500" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 38915 } }
I20260926 07:29:57.121150 25083 leader_election.cc:288] T a309a1f586c445589592c408a3e32f3e P 66291c8b70dc46a6acbfdf3c9c475500 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3a69b617c9f843b2a9001f6d598c63e8 (127.22.53.193:38223), ffbd74368f6e483abebb4ff75b582ba0 (127.22.53.194:41601)
I20260926 07:29:57.121693 24635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a309a1f586c445589592c408a3e32f3e" candidate_uuid: "66291c8b70dc46a6acbfdf3c9c475500" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a69b617c9f843b2a9001f6d598c63e8" is_pre_election: true
I20260926 07:29:57.121879 24635 raft_consensus.cc:2454] T a309a1f586c445589592c408a3e32f3e P 3a69b617c9f843b2a9001f6d598c63e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66291c8b70dc46a6acbfdf3c9c475500 in term 0.
I20260926 07:29:57.121889 24775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a309a1f586c445589592c408a3e32f3e" candidate_uuid: "66291c8b70dc46a6acbfdf3c9c475500" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffbd74368f6e483abebb4ff75b582ba0" is_pre_election: true
I20260926 07:29:57.122063 24775 raft_consensus.cc:2454] T a309a1f586c445589592c408a3e32f3e P ffbd74368f6e483abebb4ff75b582ba0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66291c8b70dc46a6acbfdf3c9c475500 in term 0.
I20260926 07:29:57.122229 24849 leader_election.cc:302] T a309a1f586c445589592c408a3e32f3e P 66291c8b70dc46a6acbfdf3c9c475500 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a69b617c9f843b2a9001f6d598c63e8, 66291c8b70dc46a6acbfdf3c9c475500; no voters:
I20260926 07:29:57.122422 25083 raft_consensus.cc:2781] T a309a1f586c445589592c408a3e32f3e P 66291c8b70dc46a6acbfdf3c9c475500 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:29:57.122473 25083 raft_consensus.cc:482] T a309a1f586c445589592c408a3e32f3e P 66291c8b70dc46a6acbfdf3c9c475500 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:29:57.122509 25083 raft_consensus.cc:3037] T a309a1f586c445589592c408a3e32f3e P 66291c8b70dc46a6acbfdf3c9c475500 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:29:57.123893 25083 raft_consensus.cc:504] T a309a1f586c445589592c408a3e32f3e P 66291c8b70dc46a6acbfdf3c9c475500 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a69b617c9f843b2a9001f6d598c63e8" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 38223 } } peers { permanent_uuid: "ffbd74368f6e483abebb4ff75b582ba0" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 41601 } } peers { permanent_uuid: "66291c8b70dc46a6acbfdf3c9c475500" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 38915 } }
I20260926 07:29:57.124305 25083 leader_election.cc:288] T a309a1f586c445589592c408a3e32f3e P 66291c8b70dc46a6acbfdf3c9c475500 [CANDIDATE]: Term 1 election: Requested vote from peers 3a69b617c9f843b2a9001f6d598c63e8 (127.22.53.193:38223), ffbd74368f6e483abebb4ff75b582ba0 (127.22.53.194:41601)
I20260926 07:29:57.124809 24635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a309a1f586c445589592c408a3e32f3e" candidate_uuid: "66291c8b70dc46a6acbfdf3c9c475500" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a69b617c9f843b2a9001f6d598c63e8"
I20260926 07:29:57.124804 24775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a309a1f586c445589592c408a3e32f3e" candidate_uuid: "66291c8b70dc46a6acbfdf3c9c475500" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffbd74368f6e483abebb4ff75b582ba0"
I20260926 07:29:57.124961 24775 raft_consensus.cc:3037] T a309a1f586c445589592c408a3e32f3e P ffbd74368f6e483abebb4ff75b582ba0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:29:57.124967 24635 raft_consensus.cc:3037] T a309a1f586c445589592c408a3e32f3e P 3a69b617c9f843b2a9001f6d598c63e8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:29:57.126021 24635 raft_consensus.cc:2454] T a309a1f586c445589592c408a3e32f3e P 3a69b617c9f843b2a9001f6d598c63e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66291c8b70dc46a6acbfdf3c9c475500 in term 1.
I20260926 07:29:57.126121 24775 raft_consensus.cc:2454] T a309a1f586c445589592c408a3e32f3e P ffbd74368f6e483abebb4ff75b582ba0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66291c8b70dc46a6acbfdf3c9c475500 in term 1.
I20260926 07:29:57.126439 24847 leader_election.cc:302] T a309a1f586c445589592c408a3e32f3e P 66291c8b70dc46a6acbfdf3c9c475500 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66291c8b70dc46a6acbfdf3c9c475500, ffbd74368f6e483abebb4ff75b582ba0; no voters:
I20260926 07:29:57.126659 25083 raft_consensus.cc:2781] T a309a1f586c445589592c408a3e32f3e P 66291c8b70dc46a6acbfdf3c9c475500 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:29:57.126753 25083 raft_consensus.cc:686] T a309a1f586c445589592c408a3e32f3e P 66291c8b70dc46a6acbfdf3c9c475500 [term 1 LEADER]: Becoming Leader. State: Replica: 66291c8b70dc46a6acbfdf3c9c475500, State: Running, Role: LEADER
I20260926 07:29:57.126920 25083 consensus_queue.cc:237] T a309a1f586c445589592c408a3e32f3e P 66291c8b70dc46a6acbfdf3c9c475500 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a69b617c9f843b2a9001f6d598c63e8" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 38223 } } peers { permanent_uuid: "ffbd74368f6e483abebb4ff75b582ba0" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 41601 } } peers { permanent_uuid: "66291c8b70dc46a6acbfdf3c9c475500" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 38915 } }
I20260926 07:29:57.128973 24474 catalog_manager.cc:5495] T a309a1f586c445589592c408a3e32f3e P 66291c8b70dc46a6acbfdf3c9c475500 reported cstate change: term changed from 0 to 1, leader changed from <none> to 66291c8b70dc46a6acbfdf3c9c475500 (127.22.53.195). New cstate: current_term: 1 leader_uuid: "66291c8b70dc46a6acbfdf3c9c475500" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a69b617c9f843b2a9001f6d598c63e8" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 38223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ffbd74368f6e483abebb4ff75b582ba0" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 41601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66291c8b70dc46a6acbfdf3c9c475500" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 38915 } health_report { overall_health: HEALTHY } } }
I20260926 07:29:57.156234 25077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:57.152941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52940 (local address 127.22.53.195:38915)
0926 07:29:57.153090 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:57.153096 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:57.153110 (+ 14us) server_negotiation.cc:407] Connection header received
0926 07:29:57.153158 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:57.153160 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:57.153204 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:57.153284 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:29:57.153534 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:57.154198 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:57.155074 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:57.155377 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:57.155486 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:57.155756 (+ 270us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:29:57.155849 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:29:57.155934 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:57.156025 (+ 91us) server_negotiation.cc:299] Negotiation successful
0926 07:29:57.156085 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 07:29:57.164484 24775 raft_consensus.cc:1261] T a309a1f586c445589592c408a3e32f3e P ffbd74368f6e483abebb4ff75b582ba0 [term 1 FOLLOWER]: Refusing update from remote peer 66291c8b70dc46a6acbfdf3c9c475500: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:29:57.164518 24635 raft_consensus.cc:1261] T a309a1f586c445589592c408a3e32f3e P 3a69b617c9f843b2a9001f6d598c63e8 [term 1 FOLLOWER]: Refusing update from remote peer 66291c8b70dc46a6acbfdf3c9c475500: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:29:57.165004 25083 consensus_queue.cc:1035] T a309a1f586c445589592c408a3e32f3e P 66291c8b70dc46a6acbfdf3c9c475500 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ffbd74368f6e483abebb4ff75b582ba0" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 41601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:29:57.165180 25039 consensus_queue.cc:1035] T a309a1f586c445589592c408a3e32f3e P 66291c8b70dc46a6acbfdf3c9c475500 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a69b617c9f843b2a9001f6d598c63e8" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 38223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 07:29:57.182680 25082 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 07:29:58.747236 24478 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47074:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:29:58.747421 24478 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47074:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:29:58.750288 24478 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 96456e5ef902415d9f99e829c7a793a1: Sending DeleteTablet for 3 replicas of tablet a309a1f586c445589592c408a3e32f3e
I20260926 07:29:58.750972 24756 tablet_service.cc:1514] Processing DeleteTablet for tablet a309a1f586c445589592c408a3e32f3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:29:58 UTC) from {username='kudu', principal='kudu/127.22.53.254@KRBTEST.COM'} at 127.0.0.1:51260
I20260926 07:29:58.751049 24896 tablet_service.cc:1514] Processing DeleteTablet for tablet a309a1f586c445589592c408a3e32f3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:29:58 UTC) from {username='kudu', principal='kudu/127.22.53.254@KRBTEST.COM'} at 127.0.0.1:52908
I20260926 07:29:58.751052 24616 tablet_service.cc:1514] Processing DeleteTablet for tablet a309a1f586c445589592c408a3e32f3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:29:58 UTC) from {username='kudu', principal='kudu/127.22.53.254@KRBTEST.COM'} at 127.0.0.1:56258
I20260926 07:29:58.751365 25135 tablet_replica.cc:330] T a309a1f586c445589592c408a3e32f3e P 66291c8b70dc46a6acbfdf3c9c475500: stopping tablet replica
I20260926 07:29:58.751387 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 24551
I20260926 07:29:58.752276 25135 raft_consensus.cc:2229] T a309a1f586c445589592c408a3e32f3e P 66291c8b70dc46a6acbfdf3c9c475500 [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:29:58.752556 25135 raft_consensus.cc:2258] T a309a1f586c445589592c408a3e32f3e P 66291c8b70dc46a6acbfdf3c9c475500 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:29:58.751474 25134 tablet_replica.cc:330] T a309a1f586c445589592c408a3e32f3e P ffbd74368f6e483abebb4ff75b582ba0: stopping tablet replica
I20260926 07:29:58.752707 25134 raft_consensus.cc:2229] T a309a1f586c445589592c408a3e32f3e P ffbd74368f6e483abebb4ff75b582ba0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:29:58.752846 25134 raft_consensus.cc:2258] T a309a1f586c445589592c408a3e32f3e P ffbd74368f6e483abebb4ff75b582ba0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:29:58.753325 25135 ts_tablet_manager.cc:1905] T a309a1f586c445589592c408a3e32f3e P 66291c8b70dc46a6acbfdf3c9c475500: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:29:58.753480 25134 ts_tablet_manager.cc:1905] T a309a1f586c445589592c408a3e32f3e P ffbd74368f6e483abebb4ff75b582ba0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:29:58.756150 25134 ts_tablet_manager.cc:1918] T a309a1f586c445589592c408a3e32f3e P ffbd74368f6e483abebb4ff75b582ba0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 07:29:58.756150 25135 ts_tablet_manager.cc:1918] T a309a1f586c445589592c408a3e32f3e P 66291c8b70dc46a6acbfdf3c9c475500: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 07:29:58.756237 25135 log.cc:1196] T a309a1f586c445589592c408a3e32f3e P 66291c8b70dc46a6acbfdf3c9c475500: Deleting WAL directory at /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407783529717-22743-0/minicluster-data/ts-2/wal/wals/a309a1f586c445589592c408a3e32f3e
I20260926 07:29:58.756237 25134 log.cc:1196] T a309a1f586c445589592c408a3e32f3e P ffbd74368f6e483abebb4ff75b582ba0: Deleting WAL directory at /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407783529717-22743-0/minicluster-data/ts-1/wal/wals/a309a1f586c445589592c408a3e32f3e
I20260926 07:29:58.756526 25135 ts_tablet_manager.cc:1939] T a309a1f586c445589592c408a3e32f3e P 66291c8b70dc46a6acbfdf3c9c475500: Deleting consensus metadata
I20260926 07:29:58.756549 25134 ts_tablet_manager.cc:1939] T a309a1f586c445589592c408a3e32f3e P ffbd74368f6e483abebb4ff75b582ba0: Deleting consensus metadata
I20260926 07:29:58.757375 24463 catalog_manager.cc:4841] TS 66291c8b70dc46a6acbfdf3c9c475500 (127.22.53.195:38915): tablet a309a1f586c445589592c408a3e32f3e (table test-table [id=2880847e76f14f228cdf31876d5d523e]) successfully deleted
I20260926 07:29:58.757596 24463 catalog_manager.cc:4841] TS ffbd74368f6e483abebb4ff75b582ba0 (127.22.53.194:41601): tablet a309a1f586c445589592c408a3e32f3e (table test-table [id=2880847e76f14f228cdf31876d5d523e]) successfully deleted
W20260926 07:29:58.761404 24465 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.53.193:38223 (error 108)
I20260926 07:29:58.761643 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 24691
I20260926 07:29:58.763466 25138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:58.762639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.193:38223 (local address 127.0.0.1:56274)
0926 07:29:58.763213 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:58.763306 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.193:38223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":85,"threads_started":1}
W20260926 07:29:58.763605 24465 catalog_manager.cc:4584] TS 3a69b617c9f843b2a9001f6d598c63e8 (127.22.53.193:38223): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a309a1f586c445589592c408a3e32f3e: Network error: Client connection negotiation failed: client connection to 127.22.53.193:38223: connect: Connection refused (error 111)
I20260926 07:29:58.769109 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 24831
I20260926 07:29:58.776710 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 24447
2026-09-26T07:29:58Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (7070 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407783529717-22743-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:29:58 dist-test-slave-wdjc krb5kdc[25141](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:29:58 dist-test-slave-wdjc krb5kdc[25141](info): set up 2 sockets
Sep 26 07:29:58 dist-test-slave-wdjc krb5kdc[25141](info): commencing operation
krb5kdc: starting...
W20260926 07:29:59.852912 22743 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407783529717-22743-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407783529717-22743-0/krb5kdc/test-user.keytab.
Sep 26 07:29:59 dist-test-slave-wdjc krb5kdc[25141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407799, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:29:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:29:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.53.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.53.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.53.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab.
Entry for principal kudu/127.22.53.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab.
I20260926 07:29:59.984309 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407783529717-22743-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407783529717-22743-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407783529717-22743-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.53.254:39741
--webserver_interface=127.22.53.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab
--principal=kudu/127.22.53.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:32839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.53.254:39741
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:30:00.107738 25154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:30:00.108143 25154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:30:00.108230 25154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:30:00.112709 25154 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:30:00.112849 25154 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:30:00.112886 25154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:30:00.112916 25154 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:30:00.112950 25154 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:30:00.118933 25154 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:32839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407783529717-22743-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407783529717-22743-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.53.254:39741
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab
--principal=kudu/127.22.53.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.53.254:39741
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.53.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25154
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407783529717-22743-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:30:00.120249 25154 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:30:00.121460 25154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:30:00.131237 25159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:00.131191 25160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:00.131263 25162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:00.131361 25154 server_base.cc:1034] running on GCE node
I20260926 07:30:00.132205 25154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:30:00.133563 25154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:30:00.134778 25154 hybrid_clock.cc:648] HybridClock initialized: now 1790407800134736 us; error 60 us; skew 500 ppm
Sep 26 07:30:00 dist-test-slave-wdjc krb5kdc[25141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407800, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:30:00.138530 25154 init.cc:377] Logged in from keytab as kudu/127.22.53.254@KRBTEST.COM (short username kudu)
I20260926 07:30:00.139525 25154 webserver.cc:460] Webserver started at http://127.22.53.254:38143/ using document root <none> and password file <none>
I20260926 07:30:00.140101 25154 fs_manager.cc:359] Metadata directory not provided
I20260926 07:30:00.140166 25154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:30:00.140341 25154 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:30:00.142715 25154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407783529717-22743-0/minicluster-data/master-0/data/instance:
uuid: "6ec3c37d743146d28dacb0f12bacf058"
format_stamp: "Formatted at 2026-09-26 07:30:00 on dist-test-slave-wdjc"
server_key: "afb1384d5f418483f31b75d8122a957b"
server_key_iv: "eb93ddd752bc03381723628809d342da"
server_key_version: "encryptionkey@0"
I20260926 07:30:00.143303 25154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407783529717-22743-0/minicluster-data/master-0/wal/instance:
uuid: "6ec3c37d743146d28dacb0f12bacf058"
format_stamp: "Formatted at 2026-09-26 07:30:00 on dist-test-slave-wdjc"
server_key: "afb1384d5f418483f31b75d8122a957b"
server_key_iv: "eb93ddd752bc03381723628809d342da"
server_key_version: "encryptionkey@0"
I20260926 07:30:00.147876 25154 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 07:30:00.150888 25169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:00.152632 25154 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:30:00.152781 25154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407783529717-22743-0/minicluster-data/master-0/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407783529717-22743-0/minicluster-data/master-0/wal
uuid: "6ec3c37d743146d28dacb0f12bacf058"
format_stamp: "Formatted at 2026-09-26 07:30:00 on dist-test-slave-wdjc"
server_key: "afb1384d5f418483f31b75d8122a957b"
server_key_iv: "eb93ddd752bc03381723628809d342da"
server_key_version: "encryptionkey@0"
I20260926 07:30:00.152911 25154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407783529717-22743-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407783529717-22743-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407783529717-22743-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:30:00.162792 25154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:30:00.166532 25154 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:30:00.166712 25154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:30:00.174762 25154 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.254:39741
I20260926 07:30:00.174794 25221 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.254:39741 every 8 connection(s)
I20260926 07:30:00.175838 25154 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
I20260926 07:30:00.178951 25222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:30:00.182044 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 25154
I20260926 07:30:00.182148 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407783529717-22743-0/minicluster-data/master-0/wal/instance
I20260926 07:30:00.182456 22743 external_mini_cluster.cc:1428] Setting key 859b1267756baea9d9315ff23800bf51
I20260926 07:30:00.185752 25222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ec3c37d743146d28dacb0f12bacf058: Bootstrap starting.
I20260926 07:30:00.188167 25222 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ec3c37d743146d28dacb0f12bacf058: Neither blocks nor log segments found. Creating new log.
I20260926 07:30:00.189069 25222 log.cc:824] T 00000000000000000000000000000000 P 6ec3c37d743146d28dacb0f12bacf058: Log is configured to *not* fsync() on all Append() calls
I20260926 07:30:00.191246 25222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ec3c37d743146d28dacb0f12bacf058: No bootstrap required, opened a new log
Sep 26 07:30:00 dist-test-slave-wdjc krb5kdc[25141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407799, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.53.254@KRBTEST.COM
I20260926 07:30:00.194971 25222 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ec3c37d743146d28dacb0f12bacf058 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ec3c37d743146d28dacb0f12bacf058" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 39741 } }
I20260926 07:30:00.195263 25222 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6ec3c37d743146d28dacb0f12bacf058 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:30:00.195365 25222 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ec3c37d743146d28dacb0f12bacf058 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ec3c37d743146d28dacb0f12bacf058, State: Initialized, Role: FOLLOWER
I20260926 07:30:00.195897 25222 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ec3c37d743146d28dacb0f12bacf058 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ec3c37d743146d28dacb0f12bacf058" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 39741 } }
I20260926 07:30:00.196120 25222 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6ec3c37d743146d28dacb0f12bacf058 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:30:00.196194 25222 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6ec3c37d743146d28dacb0f12bacf058 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:30:00.196307 25222 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ec3c37d743146d28dacb0f12bacf058 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:30:00.197408 25222 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6ec3c37d743146d28dacb0f12bacf058 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ec3c37d743146d28dacb0f12bacf058" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 39741 } }
I20260926 07:30:00.197811 25222 leader_election.cc:302] T 00000000000000000000000000000000 P 6ec3c37d743146d28dacb0f12bacf058 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6ec3c37d743146d28dacb0f12bacf058; no voters:
I20260926 07:30:00.198153 25222 leader_election.cc:288] T 00000000000000000000000000000000 P 6ec3c37d743146d28dacb0f12bacf058 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:30:00.198288 25227 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6ec3c37d743146d28dacb0f12bacf058 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:30:00.198508 25227 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6ec3c37d743146d28dacb0f12bacf058 [term 1 LEADER]: Becoming Leader. State: Replica: 6ec3c37d743146d28dacb0f12bacf058, State: Running, Role: LEADER
I20260926 07:30:00.198961 25225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:00.184070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41516 (local address 127.22.53.254:39741)
0926 07:30:00.184647 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:00.184665 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:00.184706 (+ 41us) server_negotiation.cc:407] Connection header received
0926 07:30:00.185415 (+ 709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:00.185433 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:00.185801 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:00.186208 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:00.187066 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:00.188078 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:00.189137 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:00.189694 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:00.189916 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:00.192755 (+ 2839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:00.192760 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:00.192776 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:00.192812 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:00.195208 (+ 2396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:00.195873 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:00.195879 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:00.195884 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:00.196035 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:00.196343 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:00.196351 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:00.196353 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:00.196898 (+ 545us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:00.197136 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:00.197451 (+ 315us) server_negotiation.cc:299] Negotiation successful
0926 07:30:00.197829 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":160,"threads_started":1}
I20260926 07:30:00.198904 25227 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ec3c37d743146d28dacb0f12bacf058 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ec3c37d743146d28dacb0f12bacf058" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 39741 } }
I20260926 07:30:00.199574 25222 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ec3c37d743146d28dacb0f12bacf058 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:30:00.200228 25227 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ec3c37d743146d28dacb0f12bacf058 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ec3c37d743146d28dacb0f12bacf058. Latest consensus state: current_term: 1 leader_uuid: "6ec3c37d743146d28dacb0f12bacf058" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ec3c37d743146d28dacb0f12bacf058" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 39741 } } }
I20260926 07:30:00.200357 25227 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ec3c37d743146d28dacb0f12bacf058 [sys.catalog]: This master's current role is: LEADER
I20260926 07:30:00.201514 25228 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ec3c37d743146d28dacb0f12bacf058 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6ec3c37d743146d28dacb0f12bacf058" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ec3c37d743146d28dacb0f12bacf058" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 39741 } } }
I20260926 07:30:00.201653 25228 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ec3c37d743146d28dacb0f12bacf058 [sys.catalog]: This master's current role is: LEADER
I20260926 07:30:00.202276 25233 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:30:00.205370 25233 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:30:00.211413 25233 catalog_manager.cc:1347] Generated new cluster ID: 583b99fef6614f29ac9d97eee18f8669
I20260926 07:30:00.211512 25233 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:30:00.224718 25233 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:30:00.226380 25233 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:30:00.235448 25233 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6ec3c37d743146d28dacb0f12bacf058: Generated new TSK 0
I20260926 07:30:00.236444 25233 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.53.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.53.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.53.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.193.keytab.
Entry for principal kudu/127.22.53.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.193.keytab.
I20260926 07:30:00.268306 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407783529717-22743-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407783529717-22743-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407783529717-22743-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407783529717-22743-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.53.193:0
--local_ip_for_outbound_sockets=127.22.53.193
--webserver_interface=127.22.53.193
--webserver_port=0
--tserver_master_addrs=127.22.53.254:39741
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.193.keytab
--principal=kudu/127.22.53.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:32839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:30:00.394366 25248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:30:00.394680 25248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:30:00.394757 25248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:30:00.399178 25248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:30:00.399287 25248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:30:00.399374 25248 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.53.193
I20260926 07:30:00.405417 25248 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:32839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407783529717-22743-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407783529717-22743-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.193.keytab
--principal=kudu/127.22.53.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.53.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407783529717-22743-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.53.193
--webserver_port=0
--tserver_master_addrs=127.22.53.254:39741
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25248
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.53.193
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407783529717-22743-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:30:00.406658 25248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:30:00.407779 25248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:30:00.418557 25254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:00.418572 25253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:00.418630 25248 server_base.cc:1034] running on GCE node
W20260926 07:30:00.418828 25256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:00.419237 25248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:30:00.419991 25248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:30:00.421226 25248 hybrid_clock.cc:648] HybridClock initialized: now 1790407800421209 us; error 36 us; skew 500 ppm
Sep 26 07:30:00 dist-test-slave-wdjc krb5kdc[25141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407800, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:30:00.424499 25248 init.cc:377] Logged in from keytab as kudu/127.22.53.193@KRBTEST.COM (short username kudu)
I20260926 07:30:00.425616 25248 webserver.cc:460] Webserver started at http://127.22.53.193:33039/ using document root <none> and password file <none>
I20260926 07:30:00.426249 25248 fs_manager.cc:359] Metadata directory not provided
I20260926 07:30:00.426322 25248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:30:00.426580 25248 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:30:00.428966 25248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407783529717-22743-0/minicluster-data/ts-0/data/instance:
uuid: "90aefd6b2e1044dfa345fd20107e5235"
format_stamp: "Formatted at 2026-09-26 07:30:00 on dist-test-slave-wdjc"
server_key: "0a112eba305c3bd7cb951580a3499f7c"
server_key_iv: "368a9200a08f84b0d82b3326e028195c"
server_key_version: "encryptionkey@0"
I20260926 07:30:00.429605 25248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407783529717-22743-0/minicluster-data/ts-0/wal/instance:
uuid: "90aefd6b2e1044dfa345fd20107e5235"
format_stamp: "Formatted at 2026-09-26 07:30:00 on dist-test-slave-wdjc"
server_key: "0a112eba305c3bd7cb951580a3499f7c"
server_key_iv: "368a9200a08f84b0d82b3326e028195c"
server_key_version: "encryptionkey@0"
I20260926 07:30:00.433938 25248 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260926 07:30:00.436930 25263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:00.438930 25248 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 07:30:00.439138 25248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407783529717-22743-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407783529717-22743-0/minicluster-data/ts-0/wal
uuid: "90aefd6b2e1044dfa345fd20107e5235"
format_stamp: "Formatted at 2026-09-26 07:30:00 on dist-test-slave-wdjc"
server_key: "0a112eba305c3bd7cb951580a3499f7c"
server_key_iv: "368a9200a08f84b0d82b3326e028195c"
server_key_version: "encryptionkey@0"
I20260926 07:30:00.439308 25248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407783529717-22743-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407783529717-22743-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407783529717-22743-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:30:00.460692 25248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:30:00.464843 25248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:30:00.465113 25248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:30:00.465822 25248 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:30:00.466867 25248 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:30:00.466939 25248 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:00.467015 25248 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:30:00.467067 25248 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:00.477349 25248 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.193:46245
I20260926 07:30:00.477377 25376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.193:46245 every 8 connection(s)
I20260926 07:30:00.478359 25248 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407783529717-22743-0/minicluster-data/ts-0/data/info.pb
I20260926 07:30:00.486557 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 25248
I20260926 07:30:00.486675 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407783529717-22743-0/minicluster-data/ts-0/wal/instance
I20260926 07:30:00.487226 22743 external_mini_cluster.cc:1428] Setting key 203b04901a7611fde1bf3faa8963b556
Sep 26 07:30:00 dist-test-slave-wdjc krb5kdc[25141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407800, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.193@KRBTEST.COM for kudu/127.22.53.254@KRBTEST.COM
I20260926 07:30:00.496959 25225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:00.480404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.193:39695 (local address 127.22.53.254:39741)
0926 07:30:00.480620 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:00.480625 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:00.481557 (+ 932us) server_negotiation.cc:407] Connection header received
0926 07:30:00.483050 (+ 1493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:00.483053 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:00.483121 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:00.483204 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:00.484682 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:00.485243 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:00.486563 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:00.486882 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:00.487007 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:00.490872 (+ 3865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:00.490877 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:00.490880 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:00.490927 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:00.493769 (+ 2842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:00.494547 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:00.494555 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:00.494560 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:00.494690 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:00.495229 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:00.495236 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:00.495241 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:00.495572 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:00.495699 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:00.496460 (+ 761us) server_negotiation.cc:299] Negotiation successful
0926 07:30:00.496692 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 07:30:00.497853 25379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:00.480831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:39741 (local address 127.22.53.193:39695)
0926 07:30:00.481357 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:00.481409 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:00.482738 (+ 1329us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:00.483418 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:00.483427 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:00.483863 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:00.484414 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:00.484431 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:00.485417 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:00.485421 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:00.486398 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:00.486410 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:00.487082 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:00.487087 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:00.487285 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:00.488329 (+ 1044us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:00.488355 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:00.490554 (+ 2199us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:30:00.493984 (+ 3430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:00.493993 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:00.494011 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:00.494386 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:00.494831 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:00.494834 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:00.494835 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:00.495061 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:00.495776 (+ 715us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:30:00.495783 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:30:00.496121 (+ 338us) client_negotiation.cc:769] Sending connection context
0926 07:30:00.496311 (+ 190us) client_negotiation.cc:240] Negotiation successful
0926 07:30:00.496747 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":125,"threads_started":1}
I20260926 07:30:00.499509 25377 heartbeater.cc:347] Connected to a master server at 127.22.53.254:39741
I20260926 07:30:00.499861 25377 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.22.53.194@KRBTEST.COM; defaulting to no policy
I20260926 07:30:00.500685 25377 heartbeater.cc:511] Master 127.22.53.254:39741 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.53.194@KRBTEST.COM" created.
I20260926 07:30:00.503206 25186 ts_manager.cc:194] Registered new tserver with Master: 90aefd6b2e1044dfa345fd20107e5235 (127.22.53.193:46245)
I20260926 07:30:00.505105 25186 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.53.193@KRBTEST.COM'} at 127.22.53.193:39695
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.53.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.194.keytab.
Entry for principal kudu/127.22.53.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.194.keytab.
I20260926 07:30:00.525761 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407783529717-22743-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407783529717-22743-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407783529717-22743-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407783529717-22743-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.53.194:0
--local_ip_for_outbound_sockets=127.22.53.194
--webserver_interface=127.22.53.194
--webserver_port=0
--tserver_master_addrs=127.22.53.254:39741
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.194.keytab
--principal=kudu/127.22.53.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:32839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:30:00.650543 25382 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:30:00.650928 25382 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:30:00.651014 25382 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:30:00.655606 25382 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:30:00.655738 25382 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:30:00.655843 25382 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.53.194
I20260926 07:30:00.662137 25382 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:32839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407783529717-22743-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407783529717-22743-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.194.keytab
--principal=kudu/127.22.53.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.53.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407783529717-22743-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.53.194
--webserver_port=0
--tserver_master_addrs=127.22.53.254:39741
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25382
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.53.194
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407783529717-22743-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:30:00.663374 25382 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:30:00.664448 25382 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:30:00.676043 25390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:00.676280 25387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:00.676337 25388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:00.676667 25382 server_base.cc:1034] running on GCE node
I20260926 07:30:00.677173 25382 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:30:00.677946 25382 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:30:00.679270 25382 hybrid_clock.cc:648] HybridClock initialized: now 1790407800679220 us; error 47 us; skew 500 ppm
Sep 26 07:30:00 dist-test-slave-wdjc krb5kdc[25141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407800, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:30:00.682771 25382 init.cc:377] Logged in from keytab as kudu/127.22.53.194@KRBTEST.COM (short username kudu)
I20260926 07:30:00.684051 25382 webserver.cc:460] Webserver started at http://127.22.53.194:34803/ using document root <none> and password file <none>
I20260926 07:30:00.684827 25382 fs_manager.cc:359] Metadata directory not provided
I20260926 07:30:00.684942 25382 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:30:00.685209 25382 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:30:00.687521 25382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407783529717-22743-0/minicluster-data/ts-1/data/instance:
uuid: "a7ffa93ec3eb46f29cc665937e090204"
format_stamp: "Formatted at 2026-09-26 07:30:00 on dist-test-slave-wdjc"
server_key: "ae1dfb8c12dd3aa36a2c2aa47170e436"
server_key_iv: "cfb3115266776e7602561432576f6545"
server_key_version: "encryptionkey@0"
I20260926 07:30:00.688194 25382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407783529717-22743-0/minicluster-data/ts-1/wal/instance:
uuid: "a7ffa93ec3eb46f29cc665937e090204"
format_stamp: "Formatted at 2026-09-26 07:30:00 on dist-test-slave-wdjc"
server_key: "ae1dfb8c12dd3aa36a2c2aa47170e436"
server_key_iv: "cfb3115266776e7602561432576f6545"
server_key_version: "encryptionkey@0"
I20260926 07:30:00.692953 25382 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260926 07:30:00.696785 25397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:00.698390 25382 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 07:30:00.698602 25382 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407783529717-22743-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407783529717-22743-0/minicluster-data/ts-1/wal
uuid: "a7ffa93ec3eb46f29cc665937e090204"
format_stamp: "Formatted at 2026-09-26 07:30:00 on dist-test-slave-wdjc"
server_key: "ae1dfb8c12dd3aa36a2c2aa47170e436"
server_key_iv: "cfb3115266776e7602561432576f6545"
server_key_version: "encryptionkey@0"
I20260926 07:30:00.698767 25382 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407783529717-22743-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407783529717-22743-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407783529717-22743-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:30:00.738436 25382 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:30:00.742540 25382 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:30:00.742873 25382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:30:00.743724 25382 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:30:00.744865 25382 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:30:00.744938 25382 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:00.745018 25382 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:30:00.745051 25382 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:00.754973 25382 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.194:34357
I20260926 07:30:00.755016 25510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.194:34357 every 8 connection(s)
I20260926 07:30:00.756034 25382 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407783529717-22743-0/minicluster-data/ts-1/data/info.pb
I20260926 07:30:00.764345 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 25382
I20260926 07:30:00.764456 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407783529717-22743-0/minicluster-data/ts-1/wal/instance
I20260926 07:30:00.764849 22743 external_mini_cluster.cc:1428] Setting key 8437d1a638f710894006008e5b5ace1c
Sep 26 07:30:00 dist-test-slave-wdjc krb5kdc[25141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407800, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.194@KRBTEST.COM for kudu/127.22.53.254@KRBTEST.COM
I20260926 07:30:00.774195 25225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:00.758367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.194:55919 (local address 127.22.53.254:39741)
0926 07:30:00.758550 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:00.758554 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:00.760185 (+ 1631us) server_negotiation.cc:407] Connection header received
0926 07:30:00.761617 (+ 1432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:00.761621 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:00.761704 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:00.761808 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:00.763465 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:00.764051 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:00.765814 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:00.766066 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:00.766199 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:00.769493 (+ 3294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:00.769497 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:00.769499 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:00.769531 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:00.771382 (+ 1851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:00.772209 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:00.772213 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:00.772215 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:00.772333 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:00.772846 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:00.772850 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:00.772852 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:00.773092 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:00.773215 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:00.773838 (+ 623us) server_negotiation.cc:299] Negotiation successful
0926 07:30:00.774007 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
WARNING: no policy specified for kudu/127.22.53.195@KRBTEST.COM; defaulting to no policy
I20260926 07:30:00.775264 25513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:00.758778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:39741 (local address 127.22.53.194:55919)
0926 07:30:00.759902 (+ 1124us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:00.759982 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:00.761283 (+ 1301us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:00.762100 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:00.762111 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:00.762600 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:00.763227 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:00.763243 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:00.764286 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:00.764290 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:00.765658 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:00.765672 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:00.766258 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:00.766263 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:00.766545 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:00.767301 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:00.767322 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:00.769232 (+ 1910us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:30:00.771552 (+ 2320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:00.771568 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:00.771587 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:00.771955 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:00.772458 (+ 503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:00.772461 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:00.772463 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:00.772718 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:00.773233 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:30:00.773242 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:30:00.773556 (+ 314us) client_negotiation.cc:769] Sending connection context
0926 07:30:00.773770 (+ 214us) client_negotiation.cc:240] Negotiation successful
0926 07:30:00.774205 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":792,"thread_start_us":138,"threads_started":1}
I20260926 07:30:00.777690 25511 heartbeater.cc:347] Connected to a master server at 127.22.53.254:39741
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.53.195@KRBTEST.COM" created.
I20260926 07:30:00.778155 25511 heartbeater.cc:464] Registering TS with master...
I20260926 07:30:00.779237 25511 heartbeater.cc:511] Master 127.22.53.254:39741 requested a full tablet report, sending...
I20260926 07:30:00.781592 25186 ts_manager.cc:194] Registered new tserver with Master: a7ffa93ec3eb46f29cc665937e090204 (127.22.53.194:34357)
I20260926 07:30:00.782617 25186 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.53.194@KRBTEST.COM'} at 127.22.53.194:55919
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.53.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.195.keytab.
Entry for principal kudu/127.22.53.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.195.keytab.
I20260926 07:30:00.797092 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407783529717-22743-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407783529717-22743-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407783529717-22743-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407783529717-22743-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.53.195:0
--local_ip_for_outbound_sockets=127.22.53.195
--webserver_interface=127.22.53.195
--webserver_port=0
--tserver_master_addrs=127.22.53.254:39741
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.195.keytab
--principal=kudu/127.22.53.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:32839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:30:00.922945 25516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:30:00.923273 25516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:30:00.923321 25516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:30:00.928591 25516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:30:00.928674 25516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:30:00.928745 25516 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.53.195
I20260926 07:30:00.935185 25516 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:32839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407783529717-22743-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407783529717-22743-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.195.keytab
--principal=kudu/127.22.53.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.53.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407783529717-22743-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.53.195
--webserver_port=0
--tserver_master_addrs=127.22.53.254:39741
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25516
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.53.195
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407783529717-22743-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:30:00.936715 25516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:30:00.937933 25516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:30:00.949620 25522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:00.949574 25524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:00.949597 25521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:00.949998 25516 server_base.cc:1034] running on GCE node
I20260926 07:30:00.950489 25516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:30:00.951296 25516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:30:00.952732 25516 hybrid_clock.cc:648] HybridClock initialized: now 1790407800952686 us; error 59 us; skew 500 ppm
Sep 26 07:30:00 dist-test-slave-wdjc krb5kdc[25141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407800, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:30:00.962926 25516 init.cc:377] Logged in from keytab as kudu/127.22.53.195@KRBTEST.COM (short username kudu)
I20260926 07:30:00.964076 25516 webserver.cc:460] Webserver started at http://127.22.53.195:46349/ using document root <none> and password file <none>
I20260926 07:30:00.964860 25516 fs_manager.cc:359] Metadata directory not provided
I20260926 07:30:00.964934 25516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:30:00.965122 25516 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:30:00.967382 25516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407783529717-22743-0/minicluster-data/ts-2/data/instance:
uuid: "168fea44a0514814845037315633e96d"
format_stamp: "Formatted at 2026-09-26 07:30:00 on dist-test-slave-wdjc"
server_key: "3ef0af1adc359e0244b91a50f8ad10a9"
server_key_iv: "50f3db1499a4d6576dc56a00bb8925f7"
server_key_version: "encryptionkey@0"
I20260926 07:30:00.967986 25516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407783529717-22743-0/minicluster-data/ts-2/wal/instance:
uuid: "168fea44a0514814845037315633e96d"
format_stamp: "Formatted at 2026-09-26 07:30:00 on dist-test-slave-wdjc"
server_key: "3ef0af1adc359e0244b91a50f8ad10a9"
server_key_iv: "50f3db1499a4d6576dc56a00bb8925f7"
server_key_version: "encryptionkey@0"
I20260926 07:30:00.973151 25516 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 07:30:00.976408 25531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:00.978112 25516 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:30:00.978283 25516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407783529717-22743-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407783529717-22743-0/minicluster-data/ts-2/wal
uuid: "168fea44a0514814845037315633e96d"
format_stamp: "Formatted at 2026-09-26 07:30:00 on dist-test-slave-wdjc"
server_key: "3ef0af1adc359e0244b91a50f8ad10a9"
server_key_iv: "50f3db1499a4d6576dc56a00bb8925f7"
server_key_version: "encryptionkey@0"
I20260926 07:30:00.978444 25516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407783529717-22743-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407783529717-22743-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407783529717-22743-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:30:01.008647 25516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:30:01.013768 25516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:30:01.014029 25516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:30:01.014791 25516 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:30:01.015906 25516 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:30:01.015961 25516 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:01.016005 25516 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:30:01.016052 25516 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:01.027007 25516 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.195:40469
I20260926 07:30:01.027034 25644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.195:40469 every 8 connection(s)
I20260926 07:30:01.028049 25516 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407783529717-22743-0/minicluster-data/ts-2/data/info.pb
I20260926 07:30:01.035801 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 25516
I20260926 07:30:01.035989 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407783529717-22743-0/minicluster-data/ts-2/wal/instance
I20260926 07:30:01.036334 22743 external_mini_cluster.cc:1428] Setting key 14da8530f61fb4286e93307ad2873a83
Sep 26 07:30:01 dist-test-slave-wdjc krb5kdc[25141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407800, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.195@KRBTEST.COM for kudu/127.22.53.254@KRBTEST.COM
I20260926 07:30:01.047082 25225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:01.030370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.195:60533 (local address 127.22.53.254:39741)
0926 07:30:01.030554 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:01.030559 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:01.031641 (+ 1082us) server_negotiation.cc:407] Connection header received
0926 07:30:01.033117 (+ 1476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:01.033120 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:01.033181 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:01.033282 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:01.034887 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:01.035479 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:01.037559 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:01.038014 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:01.038119 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:01.041664 (+ 3545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:01.041667 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:01.041670 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:01.041701 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:01.043471 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:01.044494 (+ 1023us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:01.044512 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:01.044514 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:01.044723 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:01.045329 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:01.045334 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:01.045335 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:01.045822 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:01.045964 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:01.046740 (+ 776us) server_negotiation.cc:299] Negotiation successful
0926 07:30:01.046919 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 07:30:01.048671 25647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:01.030837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:39741 (local address 127.22.53.195:60533)
0926 07:30:01.031411 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:01.031464 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:01.032836 (+ 1372us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:01.033571 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:01.033580 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:01.034037 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:01.034672 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:01.034690 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:01.035680 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:01.035685 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:01.037336 (+ 1651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:01.037354 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:01.038234 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:01.038238 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:01.038517 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:01.039459 (+ 942us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:01.039491 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:01.041412 (+ 1921us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:30:01.043878 (+ 2466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:01.043923 (+ 45us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:01.043940 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:01.044300 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:01.044866 (+ 566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:01.044868 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:01.044870 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:01.045091 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:01.046001 (+ 910us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:30:01.046008 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:30:01.046356 (+ 348us) client_negotiation.cc:769] Sending connection context
0926 07:30:01.046580 (+ 224us) client_negotiation.cc:240] Negotiation successful
0926 07:30:01.046933 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":144,"threads_started":1}
I20260926 07:30:01.050414 25645 heartbeater.cc:347] Connected to a master server at 127.22.53.254:39741
I20260926 07:30:01.050745 25645 heartbeater.cc:464] Registering TS with master...
I20260926 07:30:01.051424 25645 heartbeater.cc:511] Master 127.22.53.254:39741 requested a full tablet report, sending...
I20260926 07:30:01.052750 25183 ts_manager.cc:194] Registered new tserver with Master: 168fea44a0514814845037315633e96d (127.22.53.195:40469)
I20260926 07:30:01.053546 25183 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.53.195@KRBTEST.COM'} at 127.22.53.195:60533
I20260926 07:30:01.065852 22743 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 07:30:01.081290 25225 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:30:01.078985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41532 (local address 127.22.53.254:39741)
0926 07:30:01.079243 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:01.079251 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:01.079405 (+ 154us) server_negotiation.cc:407] Connection header received
0926 07:30:01.079871 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:01.079876 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:01.079953 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:01.080055 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:01.081157 (+ 1102us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41532: BlockingRecv error: recv got EOF from 127.0.0.1:41532 (error 108)
Metrics: {"server-negotiator.queue_time_us":140}
I20260926 07:30:01.083240 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 25248
I20260926 07:30:01.089355 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 25382
I20260926 07:30:01.097330 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 25516
I20260926 07:30:01.104744 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 25154
2026-09-26T07:30:01Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2328 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407783529717-22743-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:30:01 dist-test-slave-wdjc krb5kdc[25657](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:30:01 dist-test-slave-wdjc krb5kdc[25657](info): set up 2 sockets
Sep 26 07:30:01 dist-test-slave-wdjc krb5kdc[25657](info): commencing operation
krb5kdc: starting...
W20260926 07:30:02.213383 22743 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407783529717-22743-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407783529717-22743-0/krb5kdc/test-user.keytab.
Sep 26 07:30:02 dist-test-slave-wdjc krb5kdc[25657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407802, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:30:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:30:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.53.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.53.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.53.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab.
Entry for principal kudu/127.22.53.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab.
I20260926 07:30:02.357155 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407783529717-22743-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407783529717-22743-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407783529717-22743-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.53.254:45959
--webserver_interface=127.22.53.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab
--principal=kudu/127.22.53.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:36745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.53.254:45959
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:30:02.490259 25670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:30:02.490698 25670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:30:02.490782 25670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:30:02.495407 25670 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:30:02.495549 25670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:30:02.495589 25670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:30:02.495627 25670 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:30:02.495684 25670 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:30:02.501732 25670 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:36745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407783529717-22743-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407783529717-22743-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.53.254:45959
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab
--principal=kudu/127.22.53.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.53.254:45959
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.53.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25670
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407783529717-22743-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:30:02.503060 25670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:30:02.504267 25670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:30:02.514343 25676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:02.514557 25670 server_base.cc:1034] running on GCE node
W20260926 07:30:02.514343 25678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:02.514385 25675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:02.515342 25670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:30:02.516963 25670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:30:02.518190 25670 hybrid_clock.cc:648] HybridClock initialized: now 1790407802518169 us; error 47 us; skew 500 ppm
Sep 26 07:30:02 dist-test-slave-wdjc krb5kdc[25657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407802, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:30:02.521575 25670 init.cc:377] Logged in from keytab as kudu/127.22.53.254@KRBTEST.COM (short username kudu)
I20260926 07:30:02.522696 25670 webserver.cc:460] Webserver started at http://127.22.53.254:38635/ using document root <none> and password file <none>
I20260926 07:30:02.523348 25670 fs_manager.cc:359] Metadata directory not provided
I20260926 07:30:02.523422 25670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:30:02.523702 25670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:30:02.526158 25670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407783529717-22743-0/minicluster-data/master-0/data/instance:
uuid: "3a8de8c939724d0cba37722b92978866"
format_stamp: "Formatted at 2026-09-26 07:30:02 on dist-test-slave-wdjc"
server_key: "799c8c7e0a06d9c6d309d42dde818a26"
server_key_iv: "1283f071befc0a9b1f2dc41c34bb7aa6"
server_key_version: "encryptionkey@0"
I20260926 07:30:02.526873 25670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407783529717-22743-0/minicluster-data/master-0/wal/instance:
uuid: "3a8de8c939724d0cba37722b92978866"
format_stamp: "Formatted at 2026-09-26 07:30:02 on dist-test-slave-wdjc"
server_key: "799c8c7e0a06d9c6d309d42dde818a26"
server_key_iv: "1283f071befc0a9b1f2dc41c34bb7aa6"
server_key_version: "encryptionkey@0"
I20260926 07:30:02.531266 25670 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260926 07:30:02.534304 25685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:02.535840 25670 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 07:30:02.535985 25670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407783529717-22743-0/minicluster-data/master-0/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407783529717-22743-0/minicluster-data/master-0/wal
uuid: "3a8de8c939724d0cba37722b92978866"
format_stamp: "Formatted at 2026-09-26 07:30:02 on dist-test-slave-wdjc"
server_key: "799c8c7e0a06d9c6d309d42dde818a26"
server_key_iv: "1283f071befc0a9b1f2dc41c34bb7aa6"
server_key_version: "encryptionkey@0"
I20260926 07:30:02.536175 25670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407783529717-22743-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407783529717-22743-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407783529717-22743-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:30:02.553740 25670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:30:02.557808 25670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:30:02.558034 25670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:30:02.566459 25670 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.254:45959
I20260926 07:30:02.567122 25737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.254:45959 every 8 connection(s)
I20260926 07:30:02.567677 25670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
I20260926 07:30:02.571012 25738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:30:02.575681 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 25670
I20260926 07:30:02.575830 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407783529717-22743-0/minicluster-data/master-0/wal/instance
I20260926 07:30:02.576117 22743 external_mini_cluster.cc:1428] Setting key 53b6a654202cf3ecf923fe07f4aba00c
I20260926 07:30:02.578879 25738 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a8de8c939724d0cba37722b92978866: Bootstrap starting.
I20260926 07:30:02.582093 25738 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3a8de8c939724d0cba37722b92978866: Neither blocks nor log segments found. Creating new log.
I20260926 07:30:02.583114 25738 log.cc:824] T 00000000000000000000000000000000 P 3a8de8c939724d0cba37722b92978866: Log is configured to *not* fsync() on all Append() calls
I20260926 07:30:02.585937 25738 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a8de8c939724d0cba37722b92978866: No bootstrap required, opened a new log
Sep 26 07:30:02 dist-test-slave-wdjc krb5kdc[25657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407802, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.53.254@KRBTEST.COM
I20260926 07:30:02.589424 25738 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a8de8c939724d0cba37722b92978866 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a8de8c939724d0cba37722b92978866" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 45959 } }
I20260926 07:30:02.589726 25738 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3a8de8c939724d0cba37722b92978866 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:30:02.589810 25738 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a8de8c939724d0cba37722b92978866 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a8de8c939724d0cba37722b92978866, State: Initialized, Role: FOLLOWER
I20260926 07:30:02.590304 25738 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a8de8c939724d0cba37722b92978866 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a8de8c939724d0cba37722b92978866" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 45959 } }
I20260926 07:30:02.590472 25738 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3a8de8c939724d0cba37722b92978866 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:30:02.590561 25738 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3a8de8c939724d0cba37722b92978866 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:30:02.590684 25738 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a8de8c939724d0cba37722b92978866 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:30:02.591709 25738 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3a8de8c939724d0cba37722b92978866 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a8de8c939724d0cba37722b92978866" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 45959 } }
I20260926 07:30:02.592123 25738 leader_election.cc:302] T 00000000000000000000000000000000 P 3a8de8c939724d0cba37722b92978866 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3a8de8c939724d0cba37722b92978866; no voters:
I20260926 07:30:02.592592 25738 leader_election.cc:288] T 00000000000000000000000000000000 P 3a8de8c939724d0cba37722b92978866 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:30:02.592710 25743 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3a8de8c939724d0cba37722b92978866 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:30:02.592929 25743 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3a8de8c939724d0cba37722b92978866 [term 1 LEADER]: Becoming Leader. State: Replica: 3a8de8c939724d0cba37722b92978866, State: Running, Role: LEADER
I20260926 07:30:02.593271 25743 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a8de8c939724d0cba37722b92978866 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a8de8c939724d0cba37722b92978866" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 45959 } }
I20260926 07:30:02.593746 25741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:02.578128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36578 (local address 127.22.53.254:45959)
0926 07:30:02.578645 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:02.578662 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:02.578702 (+ 40us) server_negotiation.cc:407] Connection header received
0926 07:30:02.579507 (+ 805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:02.579537 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:02.579918 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:02.580245 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:02.581308 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:02.582433 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:02.583681 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:02.584360 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:02.584513 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:02.587445 (+ 2932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:02.587450 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:02.587466 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:02.587514 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:02.590147 (+ 2633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:02.590743 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:02.590750 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:02.590757 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:02.590914 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:02.591227 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:02.591231 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:02.591233 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:02.591697 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:02.591927 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:02.592238 (+ 311us) server_negotiation.cc:299] Negotiation successful
0926 07:30:02.592647 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":180,"threads_started":1}
I20260926 07:30:02.594012 25738 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a8de8c939724d0cba37722b92978866 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:30:02.594614 25743 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a8de8c939724d0cba37722b92978866 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a8de8c939724d0cba37722b92978866. Latest consensus state: current_term: 1 leader_uuid: "3a8de8c939724d0cba37722b92978866" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a8de8c939724d0cba37722b92978866" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 45959 } } }
I20260926 07:30:02.594732 25743 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a8de8c939724d0cba37722b92978866 [sys.catalog]: This master's current role is: LEADER
I20260926 07:30:02.595088 25744 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a8de8c939724d0cba37722b92978866 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3a8de8c939724d0cba37722b92978866" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a8de8c939724d0cba37722b92978866" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 45959 } } }
I20260926 07:30:02.595201 25744 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a8de8c939724d0cba37722b92978866 [sys.catalog]: This master's current role is: LEADER
I20260926 07:30:02.596091 25748 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:30:02.597282 22743 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:30:02.599560 25748 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:30:02.606494 25748 catalog_manager.cc:1347] Generated new cluster ID: 5dc00b4e057f478abd07ee134f4e7667
I20260926 07:30:02.606590 25748 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:30:02.615962 25748 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:30:02.617368 25748 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:30:02.618731 25741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:02.612391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36592 (local address 127.22.53.254:45959)
0926 07:30:02.612592 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:02.612597 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:02.612865 (+ 268us) server_negotiation.cc:407] Connection header received
0926 07:30:02.613026 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:02.613029 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:02.613088 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:02.613187 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:02.613755 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:02.614358 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:02.615610 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:02.615965 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:02.616102 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:02.617611 (+ 1509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:02.617615 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:02.617617 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:30:02.618128 (+ 511us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:02.618166 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:30:02.618184 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:02.618307 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:02.618453 (+ 146us) server_negotiation.cc:299] Negotiation successful
0926 07:30:02.618553 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260926 07:30:02.628110 25748 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3a8de8c939724d0cba37722b92978866: Generated new TSK 0
I20260926 07:30:02.629150 25748 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:30:02.635346 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 25670
2026-09-26T07:30:02Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1525 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407783529717-22743-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:30:02 dist-test-slave-wdjc krb5kdc[25771](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:30:02 dist-test-slave-wdjc krb5kdc[25771](info): set up 2 sockets
Sep 26 07:30:02 dist-test-slave-wdjc krb5kdc[25771](info): commencing operation
krb5kdc: starting...
W20260926 07:30:03.717044 22743 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407783529717-22743-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407783529717-22743-0/krb5kdc/test-user.keytab.
Sep 26 07:30:03 dist-test-slave-wdjc krb5kdc[25771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407803, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:30:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:30:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.53.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.53.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.53.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab.
Entry for principal kudu/127.22.53.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab.
I20260926 07:30:03.849229 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407783529717-22743-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407783529717-22743-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407783529717-22743-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.53.254:42429
--webserver_interface=127.22.53.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab
--principal=kudu/127.22.53.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:46859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.53.254:42429
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:30:03.975858 25784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:30:03.976269 25784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:30:03.976362 25784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:30:03.981129 25784 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:30:03.981261 25784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:30:03.981298 25784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:30:03.981340 25784 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:30:03.981375 25784 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:30:03.987504 25784 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:46859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407783529717-22743-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407783529717-22743-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.53.254:42429
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab
--principal=kudu/127.22.53.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.53.254:42429
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.53.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25784
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407783529717-22743-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:30:03.988848 25784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:30:03.990073 25784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:30:04.000481 25792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:04.000507 25790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:04.000747 25784 server_base.cc:1034] running on GCE node
W20260926 07:30:04.000918 25789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:04.001700 25784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:30:04.003150 25784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:30:04.006587 25784 hybrid_clock.cc:648] HybridClock initialized: now 1790407804006484 us; error 142 us; skew 500 ppm
Sep 26 07:30:04 dist-test-slave-wdjc krb5kdc[25771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407804, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:30:04.012256 25784 init.cc:377] Logged in from keytab as kudu/127.22.53.254@KRBTEST.COM (short username kudu)
I20260926 07:30:04.013536 25784 webserver.cc:460] Webserver started at http://127.22.53.254:39397/ using document root <none> and password file <none>
I20260926 07:30:04.014236 25784 fs_manager.cc:359] Metadata directory not provided
I20260926 07:30:04.014345 25784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:30:04.014642 25784 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:30:04.017091 25784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407783529717-22743-0/minicluster-data/master-0/data/instance:
uuid: "b0779a4fb3b44bcaa2affe49bfd4c4ea"
format_stamp: "Formatted at 2026-09-26 07:30:04 on dist-test-slave-wdjc"
server_key: "8c8291476850f66493926c6585ccc228"
server_key_iv: "1ad1c10cdcca86ab82a53342b616d68a"
server_key_version: "encryptionkey@0"
I20260926 07:30:04.017771 25784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407783529717-22743-0/minicluster-data/master-0/wal/instance:
uuid: "b0779a4fb3b44bcaa2affe49bfd4c4ea"
format_stamp: "Formatted at 2026-09-26 07:30:04 on dist-test-slave-wdjc"
server_key: "8c8291476850f66493926c6585ccc228"
server_key_iv: "1ad1c10cdcca86ab82a53342b616d68a"
server_key_version: "encryptionkey@0"
I20260926 07:30:04.022132 25784 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 07:30:04.025504 25799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:04.027225 25784 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260926 07:30:04.027429 25784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407783529717-22743-0/minicluster-data/master-0/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407783529717-22743-0/minicluster-data/master-0/wal
uuid: "b0779a4fb3b44bcaa2affe49bfd4c4ea"
format_stamp: "Formatted at 2026-09-26 07:30:04 on dist-test-slave-wdjc"
server_key: "8c8291476850f66493926c6585ccc228"
server_key_iv: "1ad1c10cdcca86ab82a53342b616d68a"
server_key_version: "encryptionkey@0"
I20260926 07:30:04.027590 25784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407783529717-22743-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407783529717-22743-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407783529717-22743-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:30:04.046093 25784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:30:04.051368 25784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:30:04.051667 25784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:30:04.061375 25784 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.254:42429
I20260926 07:30:04.061721 25851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.254:42429 every 8 connection(s)
I20260926 07:30:04.062709 25784 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
I20260926 07:30:04.066428 25852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:30:04.067976 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 25784
I20260926 07:30:04.068117 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407783529717-22743-0/minicluster-data/master-0/wal/instance
I20260926 07:30:04.068449 22743 external_mini_cluster.cc:1428] Setting key a6a8bb6d427adc4eb9b8464fafe6e802
I20260926 07:30:04.074373 25852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0779a4fb3b44bcaa2affe49bfd4c4ea: Bootstrap starting.
I20260926 07:30:04.077188 25852 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b0779a4fb3b44bcaa2affe49bfd4c4ea: Neither blocks nor log segments found. Creating new log.
I20260926 07:30:04.078207 25852 log.cc:824] T 00000000000000000000000000000000 P b0779a4fb3b44bcaa2affe49bfd4c4ea: Log is configured to *not* fsync() on all Append() calls
Sep 26 07:30:04 dist-test-slave-wdjc krb5kdc[25771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407803, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.53.254@KRBTEST.COM
I20260926 07:30:04.081640 25852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0779a4fb3b44bcaa2affe49bfd4c4ea: No bootstrap required, opened a new log
I20260926 07:30:04.087105 25852 raft_consensus.cc:348] T 00000000000000000000000000000000 P b0779a4fb3b44bcaa2affe49bfd4c4ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0779a4fb3b44bcaa2affe49bfd4c4ea" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 42429 } }
I20260926 07:30:04.088074 25852 raft_consensus.cc:374] T 00000000000000000000000000000000 P b0779a4fb3b44bcaa2affe49bfd4c4ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:30:04.088151 25855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:04.070740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56882 (local address 127.22.53.254:42429)
0926 07:30:04.071337 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:04.071353 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:04.071393 (+ 40us) server_negotiation.cc:407] Connection header received
0926 07:30:04.072360 (+ 967us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:04.072384 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:04.072907 (+ 523us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:04.073354 (+ 447us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:04.074294 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:04.075453 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:04.076430 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:04.077060 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:04.077260 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:04.081156 (+ 3896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:04.081162 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:04.081179 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:04.081214 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:04.083528 (+ 2314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:04.084108 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:04.084118 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:04.084128 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:04.084321 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:04.084862 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:04.084867 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:04.084871 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:04.085492 (+ 621us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:04.085725 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:04.086060 (+ 335us) server_negotiation.cc:299] Negotiation successful
0926 07:30:04.086436 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":185,"threads_started":1}
I20260926 07:30:04.088194 25852 raft_consensus.cc:729] T 00000000000000000000000000000000 P b0779a4fb3b44bcaa2affe49bfd4c4ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0779a4fb3b44bcaa2affe49bfd4c4ea, State: Initialized, Role: FOLLOWER
I20260926 07:30:04.088750 25852 consensus_queue.cc:260] T 00000000000000000000000000000000 P b0779a4fb3b44bcaa2affe49bfd4c4ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0779a4fb3b44bcaa2affe49bfd4c4ea" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 42429 } }
I20260926 07:30:04.088905 25852 raft_consensus.cc:388] T 00000000000000000000000000000000 P b0779a4fb3b44bcaa2affe49bfd4c4ea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:30:04.088995 25852 raft_consensus.cc:482] T 00000000000000000000000000000000 P b0779a4fb3b44bcaa2affe49bfd4c4ea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:30:04.089128 25852 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b0779a4fb3b44bcaa2affe49bfd4c4ea [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:30:04.090662 25852 raft_consensus.cc:504] T 00000000000000000000000000000000 P b0779a4fb3b44bcaa2affe49bfd4c4ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0779a4fb3b44bcaa2affe49bfd4c4ea" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 42429 } }
I20260926 07:30:04.091094 25852 leader_election.cc:302] T 00000000000000000000000000000000 P b0779a4fb3b44bcaa2affe49bfd4c4ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b0779a4fb3b44bcaa2affe49bfd4c4ea; no voters:
I20260926 07:30:04.091530 25852 leader_election.cc:288] T 00000000000000000000000000000000 P b0779a4fb3b44bcaa2affe49bfd4c4ea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:30:04.091614 25857 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b0779a4fb3b44bcaa2affe49bfd4c4ea [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:30:04.091820 25857 raft_consensus.cc:686] T 00000000000000000000000000000000 P b0779a4fb3b44bcaa2affe49bfd4c4ea [term 1 LEADER]: Becoming Leader. State: Replica: b0779a4fb3b44bcaa2affe49bfd4c4ea, State: Running, Role: LEADER
I20260926 07:30:04.092166 25857 consensus_queue.cc:237] T 00000000000000000000000000000000 P b0779a4fb3b44bcaa2affe49bfd4c4ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0779a4fb3b44bcaa2affe49bfd4c4ea" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 42429 } }
I20260926 07:30:04.093057 25852 sys_catalog.cc:564] T 00000000000000000000000000000000 P b0779a4fb3b44bcaa2affe49bfd4c4ea [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:30:04.094406 25858 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0779a4fb3b44bcaa2affe49bfd4c4ea [sys.catalog]: SysCatalogTable state changed. Reason: New leader b0779a4fb3b44bcaa2affe49bfd4c4ea. Latest consensus state: current_term: 1 leader_uuid: "b0779a4fb3b44bcaa2affe49bfd4c4ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0779a4fb3b44bcaa2affe49bfd4c4ea" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 42429 } } }
I20260926 07:30:04.094589 25858 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0779a4fb3b44bcaa2affe49bfd4c4ea [sys.catalog]: This master's current role is: LEADER
I20260926 07:30:04.095001 25859 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0779a4fb3b44bcaa2affe49bfd4c4ea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b0779a4fb3b44bcaa2affe49bfd4c4ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0779a4fb3b44bcaa2affe49bfd4c4ea" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 42429 } } }
I20260926 07:30:04.095124 25859 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0779a4fb3b44bcaa2affe49bfd4c4ea [sys.catalog]: This master's current role is: LEADER
I20260926 07:30:04.095467 25866 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:30:04.099032 25866 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:30:04.106245 25866 catalog_manager.cc:1347] Generated new cluster ID: 5f97f3b027c640e9b89948730cb5c7a6
I20260926 07:30:04.106360 25866 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:30:04.134616 25866 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:30:04.135735 25866 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:30:04.146796 25866 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b0779a4fb3b44bcaa2affe49bfd4c4ea: Generated new TSK 0
I20260926 07:30:04.147918 25866 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.53.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.53.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.53.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.193.keytab.
Entry for principal kudu/127.22.53.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.193.keytab.
I20260926 07:30:04.195974 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407783529717-22743-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407783529717-22743-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407783529717-22743-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407783529717-22743-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.53.193:0
--local_ip_for_outbound_sockets=127.22.53.193
--webserver_interface=127.22.53.193
--webserver_port=0
--tserver_master_addrs=127.22.53.254:42429
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.193.keytab
--principal=kudu/127.22.53.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:46859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:30:04.333839 25878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:30:04.334215 25878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:30:04.334297 25878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:30:04.338717 25878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:30:04.338852 25878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:30:04.338965 25878 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.53.193
I20260926 07:30:04.345050 25878 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:46859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407783529717-22743-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407783529717-22743-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.193.keytab
--principal=kudu/127.22.53.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.53.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407783529717-22743-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.53.193
--webserver_port=0
--tserver_master_addrs=127.22.53.254:42429
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25878
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.53.193
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407783529717-22743-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:30:04.346374 25878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:30:04.347615 25878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:30:04.360445 25883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:04.360623 25884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:04.360623 25886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:04.361006 25878 server_base.cc:1034] running on GCE node
I20260926 07:30:04.361488 25878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:30:04.362252 25878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:30:04.363571 25878 hybrid_clock.cc:648] HybridClock initialized: now 1790407804363533 us; error 38 us; skew 500 ppm
Sep 26 07:30:04 dist-test-slave-wdjc krb5kdc[25771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407804, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:30:04.367452 25878 init.cc:377] Logged in from keytab as kudu/127.22.53.193@KRBTEST.COM (short username kudu)
I20260926 07:30:04.368520 25878 webserver.cc:460] Webserver started at http://127.22.53.193:46035/ using document root <none> and password file <none>
I20260926 07:30:04.369227 25878 fs_manager.cc:359] Metadata directory not provided
I20260926 07:30:04.369295 25878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:30:04.369485 25878 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:30:04.371764 25878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407783529717-22743-0/minicluster-data/ts-0/data/instance:
uuid: "c96fd9cd805940459a5a14c914b858d3"
format_stamp: "Formatted at 2026-09-26 07:30:04 on dist-test-slave-wdjc"
server_key: "7e93d8ecb5df2f72d81988a45cf523f6"
server_key_iv: "42ad32bbc6ca3989fadada581d78829a"
server_key_version: "encryptionkey@0"
I20260926 07:30:04.372360 25878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407783529717-22743-0/minicluster-data/ts-0/wal/instance:
uuid: "c96fd9cd805940459a5a14c914b858d3"
format_stamp: "Formatted at 2026-09-26 07:30:04 on dist-test-slave-wdjc"
server_key: "7e93d8ecb5df2f72d81988a45cf523f6"
server_key_iv: "42ad32bbc6ca3989fadada581d78829a"
server_key_version: "encryptionkey@0"
I20260926 07:30:04.376840 25878 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 07:30:04.379722 25893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:04.381436 25878 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:30:04.381609 25878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407783529717-22743-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407783529717-22743-0/minicluster-data/ts-0/wal
uuid: "c96fd9cd805940459a5a14c914b858d3"
format_stamp: "Formatted at 2026-09-26 07:30:04 on dist-test-slave-wdjc"
server_key: "7e93d8ecb5df2f72d81988a45cf523f6"
server_key_iv: "42ad32bbc6ca3989fadada581d78829a"
server_key_version: "encryptionkey@0"
I20260926 07:30:04.381754 25878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407783529717-22743-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407783529717-22743-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407783529717-22743-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:30:04.406912 25878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:30:04.411026 25878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:30:04.411307 25878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:30:04.412017 25878 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:30:04.413101 25878 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:30:04.413182 25878 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:04.413261 25878 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:30:04.413313 25878 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:04.423687 25878 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.193:45645
I20260926 07:30:04.423709 26006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.193:45645 every 8 connection(s)
I20260926 07:30:04.424850 25878 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407783529717-22743-0/minicluster-data/ts-0/data/info.pb
I20260926 07:30:04.435075 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 25878
I20260926 07:30:04.435266 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407783529717-22743-0/minicluster-data/ts-0/wal/instance
I20260926 07:30:04.435596 22743 external_mini_cluster.cc:1428] Setting key 54b9f2c69ff50558f233a28e76df09dc
Sep 26 07:30:04 dist-test-slave-wdjc krb5kdc[25771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407804, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.193@KRBTEST.COM for kudu/127.22.53.254@KRBTEST.COM
I20260926 07:30:04.443704 25855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:04.427127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.193:59009 (local address 127.22.53.254:42429)
0926 07:30:04.427287 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:04.427291 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:04.428284 (+ 993us) server_negotiation.cc:407] Connection header received
0926 07:30:04.429889 (+ 1605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:04.429893 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:04.429959 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:04.430388 (+ 429us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:04.432760 (+ 2372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:04.433350 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:04.434761 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:04.435018 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:04.435150 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:04.438813 (+ 3663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:04.438816 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:04.438818 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:04.438845 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:04.440813 (+ 1968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:04.441579 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:04.441586 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:04.441591 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:04.441705 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:04.442112 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:04.442118 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:04.442123 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:04.442414 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:04.442532 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:04.443254 (+ 722us) server_negotiation.cc:299] Negotiation successful
0926 07:30:04.443421 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 07:30:04.444509 26009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:04.427557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:42429 (local address 127.22.53.193:59009)
0926 07:30:04.428086 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:04.428136 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:04.429628 (+ 1492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:04.430716 (+ 1088us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:04.430729 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:04.431400 (+ 671us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:04.432385 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:04.432469 (+ 84us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:04.433533 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:04.433542 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:04.434587 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:04.434599 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:04.435210 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:04.435215 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:04.435466 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:04.436267 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:04.436294 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:04.438554 (+ 2260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:30:04.441012 (+ 2458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:04.441019 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:04.441036 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:04.441400 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:04.441812 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:04.441815 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:04.441816 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:04.441982 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:04.442531 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:30:04.442537 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:30:04.442930 (+ 393us) client_negotiation.cc:769] Sending connection context
0926 07:30:04.443139 (+ 209us) client_negotiation.cc:240] Negotiation successful
0926 07:30:04.443466 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":133,"threads_started":1}
I20260926 07:30:04.446125 26007 heartbeater.cc:347] Connected to a master server at 127.22.53.254:42429
I20260926 07:30:04.446417 26007 heartbeater.cc:464] Registering TS with master...
I20260926 07:30:04.447024 26007 heartbeater.cc:511] Master 127.22.53.254:42429 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.22.53.194@KRBTEST.COM; defaulting to no policy
I20260926 07:30:04.448939 25816 ts_manager.cc:194] Registered new tserver with Master: c96fd9cd805940459a5a14c914b858d3 (127.22.53.193:45645)
I20260926 07:30:04.450294 25816 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.53.193@KRBTEST.COM'} at 127.22.53.193:59009
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.53.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.53.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.194.keytab.
Entry for principal kudu/127.22.53.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.194.keytab.
I20260926 07:30:04.468992 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407783529717-22743-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407783529717-22743-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407783529717-22743-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407783529717-22743-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.53.194:0
--local_ip_for_outbound_sockets=127.22.53.194
--webserver_interface=127.22.53.194
--webserver_port=0
--tserver_master_addrs=127.22.53.254:42429
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.194.keytab
--principal=kudu/127.22.53.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:46859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:30:04.591113 26012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:30:04.591446 26012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:30:04.591495 26012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:30:04.596469 26012 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:30:04.596606 26012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:30:04.596705 26012 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.53.194
I20260926 07:30:04.602608 26012 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:46859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407783529717-22743-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407783529717-22743-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.194.keytab
--principal=kudu/127.22.53.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.53.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407783529717-22743-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.53.194
--webserver_port=0
--tserver_master_addrs=127.22.53.254:42429
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26012
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.53.194
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407783529717-22743-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:30:04.603858 26012 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:30:04.605075 26012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:30:04.616031 26017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:04.616058 26018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:04.616158 26020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:04.616294 26012 server_base.cc:1034] running on GCE node
I20260926 07:30:04.616874 26012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:30:04.617615 26012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:30:04.618856 26012 hybrid_clock.cc:648] HybridClock initialized: now 1790407804618841 us; error 28 us; skew 500 ppm
Sep 26 07:30:04 dist-test-slave-wdjc krb5kdc[25771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407804, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:30:04.621935 26012 init.cc:377] Logged in from keytab as kudu/127.22.53.194@KRBTEST.COM (short username kudu)
I20260926 07:30:04.622952 26012 webserver.cc:460] Webserver started at http://127.22.53.194:34051/ using document root <none> and password file <none>
I20260926 07:30:04.623606 26012 fs_manager.cc:359] Metadata directory not provided
I20260926 07:30:04.623708 26012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:30:04.623962 26012 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:30:04.626349 26012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407783529717-22743-0/minicluster-data/ts-1/data/instance:
uuid: "91775bac0d63499092f8a04f0c05a273"
format_stamp: "Formatted at 2026-09-26 07:30:04 on dist-test-slave-wdjc"
server_key: "ea7f44458ef0da837ddbb216b3849e7f"
server_key_iv: "2d1f053e2f401c01c77e77491e6c387d"
server_key_version: "encryptionkey@0"
I20260926 07:30:04.626992 26012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407783529717-22743-0/minicluster-data/ts-1/wal/instance:
uuid: "91775bac0d63499092f8a04f0c05a273"
format_stamp: "Formatted at 2026-09-26 07:30:04 on dist-test-slave-wdjc"
server_key: "ea7f44458ef0da837ddbb216b3849e7f"
server_key_iv: "2d1f053e2f401c01c77e77491e6c387d"
server_key_version: "encryptionkey@0"
I20260926 07:30:04.631321 26012 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 07:30:04.634619 26027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:04.636332 26012 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 07:30:04.636584 26012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407783529717-22743-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407783529717-22743-0/minicluster-data/ts-1/wal
uuid: "91775bac0d63499092f8a04f0c05a273"
format_stamp: "Formatted at 2026-09-26 07:30:04 on dist-test-slave-wdjc"
server_key: "ea7f44458ef0da837ddbb216b3849e7f"
server_key_iv: "2d1f053e2f401c01c77e77491e6c387d"
server_key_version: "encryptionkey@0"
I20260926 07:30:04.636750 26012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407783529717-22743-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407783529717-22743-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407783529717-22743-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:30:04.652279 26012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:30:04.656745 26012 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:30:04.656996 26012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:30:04.657665 26012 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:30:04.658577 26012 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:30:04.658655 26012 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:04.658727 26012 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:30:04.658782 26012 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:04.668763 26012 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.194:43669
I20260926 07:30:04.668780 26140 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.194:43669 every 8 connection(s)
I20260926 07:30:04.670003 26012 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407783529717-22743-0/minicluster-data/ts-1/data/info.pb
I20260926 07:30:04.677155 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 26012
I20260926 07:30:04.677304 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407783529717-22743-0/minicluster-data/ts-1/wal/instance
I20260926 07:30:04.677585 22743 external_mini_cluster.cc:1428] Setting key c0556e6fa4daf0a957f1983c99aeb455
Sep 26 07:30:04 dist-test-slave-wdjc krb5kdc[25771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407804, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.194@KRBTEST.COM for kudu/127.22.53.254@KRBTEST.COM
I20260926 07:30:04.686343 25855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:04.672257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.194:39775 (local address 127.22.53.254:42429)
0926 07:30:04.672441 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:04.672445 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:04.673408 (+ 963us) server_negotiation.cc:407] Connection header received
0926 07:30:04.674973 (+ 1565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:04.674977 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:04.675043 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:04.675154 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:04.676628 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:04.677191 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:04.678680 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:04.678933 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:04.679052 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:04.681997 (+ 2945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:04.682001 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:04.682004 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:04.682044 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:04.683726 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:04.684378 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:04.684382 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:04.684384 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:04.684474 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:04.684962 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:04.684966 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:04.684968 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:04.685184 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:04.685300 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:04.685955 (+ 655us) server_negotiation.cc:299] Negotiation successful
0926 07:30:04.686163 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
WARNING: no policy specified for kudu/127.22.53.195@KRBTEST.COM; defaulting to no policy
I20260926 07:30:04.687438 26143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:04.672716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:42429 (local address 127.22.53.194:39775)
0926 07:30:04.673245 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:04.673291 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:04.674674 (+ 1383us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:04.675357 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:04.675364 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:04.675875 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:04.676430 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:04.676445 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:04.677339 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:04.677343 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:04.678544 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:04.678556 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:04.679102 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:04.679106 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:04.679288 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:04.679969 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:04.679988 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:04.681650 (+ 1662us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:30:04.683889 (+ 2239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:04.683896 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:04.683921 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:04.684254 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:04.684672 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:04.684676 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:04.684678 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:04.684848 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:04.685298 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:30:04.685308 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:30:04.685669 (+ 361us) client_negotiation.cc:769] Sending connection context
0926 07:30:04.685899 (+ 230us) client_negotiation.cc:240] Negotiation successful
0926 07:30:04.686297 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":134,"threads_started":1}
I20260926 07:30:04.688843 26141 heartbeater.cc:347] Connected to a master server at 127.22.53.254:42429
I20260926 07:30:04.689129 26141 heartbeater.cc:464] Registering TS with master...
I20260926 07:30:04.689725 26141 heartbeater.cc:511] Master 127.22.53.254:42429 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.53.195@KRBTEST.COM" created.
I20260926 07:30:04.691040 25816 ts_manager.cc:194] Registered new tserver with Master: 91775bac0d63499092f8a04f0c05a273 (127.22.53.194:43669)
I20260926 07:30:04.692082 25816 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.53.194@KRBTEST.COM'} at 127.22.53.194:39775
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.53.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.195.keytab.
Entry for principal kudu/127.22.53.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.195.keytab.
I20260926 07:30:04.708415 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407783529717-22743-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407783529717-22743-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407783529717-22743-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407783529717-22743-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.53.195:0
--local_ip_for_outbound_sockets=127.22.53.195
--webserver_interface=127.22.53.195
--webserver_port=0
--tserver_master_addrs=127.22.53.254:42429
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.195.keytab
--principal=kudu/127.22.53.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:46859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:30:04.834374 26146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:30:04.834733 26146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:30:04.834841 26146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:30:04.839515 26146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:30:04.839636 26146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:30:04.839709 26146 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.53.195
I20260926 07:30:04.846040 26146 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:46859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407783529717-22743-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407783529717-22743-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.195.keytab
--principal=kudu/127.22.53.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.53.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407783529717-22743-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.53.195
--webserver_port=0
--tserver_master_addrs=127.22.53.254:42429
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26146
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.53.195
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407783529717-22743-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:30:04.847234 26146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:30:04.848351 26146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:30:04.859998 26152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:04.860008 26151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:04.860033 26154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:04.860584 26146 server_base.cc:1034] running on GCE node
I20260926 07:30:04.861157 26146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:30:04.861996 26146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:30:04.863269 26146 hybrid_clock.cc:648] HybridClock initialized: now 1790407804863229 us; error 58 us; skew 500 ppm
Sep 26 07:30:04 dist-test-slave-wdjc krb5kdc[25771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407804, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:30:04.866595 26146 init.cc:377] Logged in from keytab as kudu/127.22.53.195@KRBTEST.COM (short username kudu)
I20260926 07:30:04.867961 26146 webserver.cc:460] Webserver started at http://127.22.53.195:39781/ using document root <none> and password file <none>
I20260926 07:30:04.868774 26146 fs_manager.cc:359] Metadata directory not provided
I20260926 07:30:04.868891 26146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:30:04.869181 26146 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:30:04.871632 26146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407783529717-22743-0/minicluster-data/ts-2/data/instance:
uuid: "dc6d47c8ff494e2599dd3897b656ef4e"
format_stamp: "Formatted at 2026-09-26 07:30:04 on dist-test-slave-wdjc"
server_key: "22e4890411ed811b5a7209f73f8b1514"
server_key_iv: "d37b737395e21894835601c4f0cefe14"
server_key_version: "encryptionkey@0"
I20260926 07:30:04.872315 26146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407783529717-22743-0/minicluster-data/ts-2/wal/instance:
uuid: "dc6d47c8ff494e2599dd3897b656ef4e"
format_stamp: "Formatted at 2026-09-26 07:30:04 on dist-test-slave-wdjc"
server_key: "22e4890411ed811b5a7209f73f8b1514"
server_key_iv: "d37b737395e21894835601c4f0cefe14"
server_key_version: "encryptionkey@0"
I20260926 07:30:04.877036 26146 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260926 07:30:04.880385 26161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:04.882469 26146 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 07:30:04.882669 26146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407783529717-22743-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407783529717-22743-0/minicluster-data/ts-2/wal
uuid: "dc6d47c8ff494e2599dd3897b656ef4e"
format_stamp: "Formatted at 2026-09-26 07:30:04 on dist-test-slave-wdjc"
server_key: "22e4890411ed811b5a7209f73f8b1514"
server_key_iv: "d37b737395e21894835601c4f0cefe14"
server_key_version: "encryptionkey@0"
I20260926 07:30:04.882823 26146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407783529717-22743-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407783529717-22743-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407783529717-22743-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:30:04.913864 26146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:30:04.917788 26146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:30:04.918043 26146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:30:04.918741 26146 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:30:04.919685 26146 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:30:04.919761 26146 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:04.919836 26146 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:30:04.919888 26146 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:04.930114 26146 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.195:45675
I20260926 07:30:04.930137 26274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.195:45675 every 8 connection(s)
I20260926 07:30:04.931149 26146 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407783529717-22743-0/minicluster-data/ts-2/data/info.pb
I20260926 07:30:04.937085 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 26146
I20260926 07:30:04.937250 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407783529717-22743-0/minicluster-data/ts-2/wal/instance
I20260926 07:30:04.937578 22743 external_mini_cluster.cc:1428] Setting key 08cea32e3bc7ab31705823dd15a13f3e
Sep 26 07:30:04 dist-test-slave-wdjc krb5kdc[25771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407804, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.195@KRBTEST.COM for kudu/127.22.53.254@KRBTEST.COM
I20260926 07:30:04.950647 25855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:04.933350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.195:47443 (local address 127.22.53.254:42429)
0926 07:30:04.933570 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:04.933575 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:04.934924 (+ 1349us) server_negotiation.cc:407] Connection header received
0926 07:30:04.937055 (+ 2131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:04.937061 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:04.937147 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:04.937274 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:04.939116 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:04.939924 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:04.941450 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:04.941695 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:04.941783 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:04.945087 (+ 3304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:04.945091 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:04.945093 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:04.945123 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:04.947070 (+ 1947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:04.948153 (+ 1083us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:04.948156 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:04.948158 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:04.948289 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:04.949080 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:04.949087 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:04.949088 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:04.949401 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:04.949522 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:04.950306 (+ 784us) server_negotiation.cc:299] Negotiation successful
0926 07:30:04.950501 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260926 07:30:04.951869 26277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:04.933829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:42429 (local address 127.22.53.195:47443)
0926 07:30:04.934651 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:04.934743 (+ 92us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:04.936702 (+ 1959us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:04.937653 (+ 951us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:04.937670 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:04.938135 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:04.938950 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:04.938970 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:04.940099 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:04.940103 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:04.941299 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:04.941312 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:04.942000 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:04.942006 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:04.942365 (+ 359us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:04.943103 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:04.943122 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:04.944809 (+ 1687us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:30:04.947327 (+ 2518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:04.947337 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:04.947364 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:04.947906 (+ 542us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:04.948460 (+ 554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:04.948464 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:04.948466 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:04.948879 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:04.949551 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:30:04.949563 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:30:04.949944 (+ 381us) client_negotiation.cc:769] Sending connection context
0926 07:30:04.950153 (+ 209us) client_negotiation.cc:240] Negotiation successful
0926 07:30:04.950578 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":138,"threads_started":1}
I20260926 07:30:04.953227 26275 heartbeater.cc:347] Connected to a master server at 127.22.53.254:42429
I20260926 07:30:04.953537 26275 heartbeater.cc:464] Registering TS with master...
I20260926 07:30:04.954150 26275 heartbeater.cc:511] Master 127.22.53.254:42429 requested a full tablet report, sending...
I20260926 07:30:04.955370 25816 ts_manager.cc:194] Registered new tserver with Master: dc6d47c8ff494e2599dd3897b656ef4e (127.22.53.195:45675)
I20260926 07:30:04.956046 25816 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.53.195@KRBTEST.COM'} at 127.22.53.195:47443
I20260926 07:30:04.965709 22743 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 07:30:04 dist-test-slave-wdjc krb5kdc[25771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407804, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 26 07:30:04 dist-test-slave-wdjc krb5kdc[25771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407804, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.22.53.254@KRBTEST.COM
I20260926 07:30:04.995749 25855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:04.985835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56888 (local address 127.22.53.254:42429)
0926 07:30:04.986050 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:04.986057 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:04.986169 (+ 112us) server_negotiation.cc:407] Connection header received
0926 07:30:04.986352 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:04.986356 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:04.986411 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:04.986507 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:04.987513 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:04.988070 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:04.989100 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:04.989328 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:04.989442 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:04.992293 (+ 2851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:04.992296 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:04.992298 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:04.992331 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:04.993835 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:04.994305 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:04.994310 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:04.994312 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:04.994418 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:04.994738 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:04.994741 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:04.994742 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:04.994995 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:04.995104 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:04.995430 (+ 326us) server_negotiation.cc:299] Negotiation successful
0926 07:30:04.995618 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
W20260926 07:30:04.996217 25816 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:56888
I20260926 07:30:04.997431 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 25878
I20260926 07:30:05.003960 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 26012
I20260926 07:30:05.010349 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 26146
I20260926 07:30:05.017271 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 25784
2026-09-26T07:30:05Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2386 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407783529717-22743-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:30:05 dist-test-slave-wdjc krb5kdc[26287](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:30:05 dist-test-slave-wdjc krb5kdc[26287](info): set up 2 sockets
Sep 26 07:30:05 dist-test-slave-wdjc krb5kdc[26287](info): commencing operation
krb5kdc: starting...
W20260926 07:30:06.113359 22743 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407783529717-22743-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407783529717-22743-0/krb5kdc/test-user.keytab.
Sep 26 07:30:06 dist-test-slave-wdjc krb5kdc[26287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407806, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:30:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:30:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.53.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.53.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.53.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab.
Entry for principal kudu/127.22.53.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab.
I20260926 07:30:06.274156 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407783529717-22743-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407783529717-22743-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407783529717-22743-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.53.254:44799
--webserver_interface=127.22.53.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab
--principal=kudu/127.22.53.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:37411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.53.254:44799
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:30:06.399962 26300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:30:06.400434 26300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:30:06.400593 26300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:30:06.406870 26300 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:30:06.407001 26300 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:30:06.407038 26300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:30:06.407080 26300 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:30:06.407119 26300 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:30:06.413532 26300 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:37411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407783529717-22743-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407783529717-22743-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.53.254:44799
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab
--principal=kudu/127.22.53.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.53.254:44799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.53.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26300
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407783529717-22743-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:30:06.414819 26300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:30:06.415988 26300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:30:06.425916 26305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:06.425907 26306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:06.426076 26300 server_base.cc:1034] running on GCE node
W20260926 07:30:06.426348 26308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:06.427016 26300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:30:06.428408 26300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:30:06.429684 26300 hybrid_clock.cc:648] HybridClock initialized: now 1790407806429637 us; error 68 us; skew 500 ppm
Sep 26 07:30:06 dist-test-slave-wdjc krb5kdc[26287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407806, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:30:06.436906 26300 init.cc:377] Logged in from keytab as kudu/127.22.53.254@KRBTEST.COM (short username kudu)
I20260926 07:30:06.438161 26300 webserver.cc:460] Webserver started at http://127.22.53.254:43519/ using document root <none> and password file <none>
I20260926 07:30:06.438818 26300 fs_manager.cc:359] Metadata directory not provided
I20260926 07:30:06.438925 26300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:30:06.439182 26300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:30:06.441608 26300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407783529717-22743-0/minicluster-data/master-0/data/instance:
uuid: "22b70b0546574e2f8a4a3a3962a82b45"
format_stamp: "Formatted at 2026-09-26 07:30:06 on dist-test-slave-wdjc"
server_key: "b6c5f99b89ddb7faafd86e6b6c32ef6f"
server_key_iv: "fa2ab5fa51e9fe1dc73d90be2deb8425"
server_key_version: "encryptionkey@0"
I20260926 07:30:06.442308 26300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407783529717-22743-0/minicluster-data/master-0/wal/instance:
uuid: "22b70b0546574e2f8a4a3a3962a82b45"
format_stamp: "Formatted at 2026-09-26 07:30:06 on dist-test-slave-wdjc"
server_key: "b6c5f99b89ddb7faafd86e6b6c32ef6f"
server_key_iv: "fa2ab5fa51e9fe1dc73d90be2deb8425"
server_key_version: "encryptionkey@0"
I20260926 07:30:06.446653 26300 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 07:30:06.449649 26315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:06.451110 26300 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 07:30:06.451242 26300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407783529717-22743-0/minicluster-data/master-0/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407783529717-22743-0/minicluster-data/master-0/wal
uuid: "22b70b0546574e2f8a4a3a3962a82b45"
format_stamp: "Formatted at 2026-09-26 07:30:06 on dist-test-slave-wdjc"
server_key: "b6c5f99b89ddb7faafd86e6b6c32ef6f"
server_key_iv: "fa2ab5fa51e9fe1dc73d90be2deb8425"
server_key_version: "encryptionkey@0"
I20260926 07:30:06.451419 26300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407783529717-22743-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407783529717-22743-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407783529717-22743-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:30:06.480384 26300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:30:06.484743 26300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:30:06.484999 26300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:30:06.494141 26300 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.254:44799
I20260926 07:30:06.494155 26367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.254:44799 every 8 connection(s)
I20260926 07:30:06.495340 26300 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
I20260926 07:30:06.498744 26368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:30:06.503532 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 26300
I20260926 07:30:06.503682 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407783529717-22743-0/minicluster-data/master-0/wal/instance
I20260926 07:30:06.503952 22743 external_mini_cluster.cc:1428] Setting key 9cefd3b1a3f79dd085f244414618c545
I20260926 07:30:06.506007 26368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22b70b0546574e2f8a4a3a3962a82b45: Bootstrap starting.
I20260926 07:30:06.508649 26368 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 22b70b0546574e2f8a4a3a3962a82b45: Neither blocks nor log segments found. Creating new log.
I20260926 07:30:06.509549 26368 log.cc:824] T 00000000000000000000000000000000 P 22b70b0546574e2f8a4a3a3962a82b45: Log is configured to *not* fsync() on all Append() calls
I20260926 07:30:06.511989 26368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22b70b0546574e2f8a4a3a3962a82b45: No bootstrap required, opened a new log
Sep 26 07:30:06 dist-test-slave-wdjc krb5kdc[26287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407806, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.53.254@KRBTEST.COM
I20260926 07:30:06.516480 26368 raft_consensus.cc:348] T 00000000000000000000000000000000 P 22b70b0546574e2f8a4a3a3962a82b45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22b70b0546574e2f8a4a3a3962a82b45" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 44799 } }
I20260926 07:30:06.516935 26368 raft_consensus.cc:374] T 00000000000000000000000000000000 P 22b70b0546574e2f8a4a3a3962a82b45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:30:06.517040 26368 raft_consensus.cc:729] T 00000000000000000000000000000000 P 22b70b0546574e2f8a4a3a3962a82b45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22b70b0546574e2f8a4a3a3962a82b45, State: Initialized, Role: FOLLOWER
I20260926 07:30:06.517570 26368 consensus_queue.cc:260] T 00000000000000000000000000000000 P 22b70b0546574e2f8a4a3a3962a82b45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22b70b0546574e2f8a4a3a3962a82b45" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 44799 } }
I20260926 07:30:06.517741 26368 raft_consensus.cc:388] T 00000000000000000000000000000000 P 22b70b0546574e2f8a4a3a3962a82b45 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:30:06.517832 26368 raft_consensus.cc:482] T 00000000000000000000000000000000 P 22b70b0546574e2f8a4a3a3962a82b45 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:30:06.517966 26368 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 22b70b0546574e2f8a4a3a3962a82b45 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:30:06.519295 26368 raft_consensus.cc:504] T 00000000000000000000000000000000 P 22b70b0546574e2f8a4a3a3962a82b45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22b70b0546574e2f8a4a3a3962a82b45" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 44799 } }
I20260926 07:30:06.519765 26368 leader_election.cc:302] T 00000000000000000000000000000000 P 22b70b0546574e2f8a4a3a3962a82b45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 22b70b0546574e2f8a4a3a3962a82b45; no voters:
I20260926 07:30:06.520160 26368 leader_election.cc:288] T 00000000000000000000000000000000 P 22b70b0546574e2f8a4a3a3962a82b45 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:30:06.520334 26373 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 22b70b0546574e2f8a4a3a3962a82b45 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:30:06.520853 26373 raft_consensus.cc:686] T 00000000000000000000000000000000 P 22b70b0546574e2f8a4a3a3962a82b45 [term 1 LEADER]: Becoming Leader. State: Replica: 22b70b0546574e2f8a4a3a3962a82b45, State: Running, Role: LEADER
I20260926 07:30:06.521227 26373 consensus_queue.cc:237] T 00000000000000000000000000000000 P 22b70b0546574e2f8a4a3a3962a82b45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22b70b0546574e2f8a4a3a3962a82b45" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 44799 } }
I20260926 07:30:06.521682 26368 sys_catalog.cc:564] T 00000000000000000000000000000000 P 22b70b0546574e2f8a4a3a3962a82b45 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:30:06.522214 26371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:06.505728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60618 (local address 127.22.53.254:44799)
0926 07:30:06.506338 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:06.506356 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:06.506398 (+ 42us) server_negotiation.cc:407] Connection header received
0926 07:30:06.507173 (+ 775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:06.507193 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:06.507589 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:06.507994 (+ 405us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:06.509102 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:06.510535 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:06.511844 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:06.512512 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:06.512777 (+ 265us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:06.515595 (+ 2818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:06.515601 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:06.515619 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:06.515658 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:06.517846 (+ 2188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:06.518620 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:06.518629 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:06.518638 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:06.518930 (+ 292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:06.519451 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:06.519458 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:06.519463 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:06.520100 (+ 637us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:06.520358 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:06.520812 (+ 454us) server_negotiation.cc:299] Negotiation successful
0926 07:30:06.521173 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":180,"threads_started":1}
I20260926 07:30:06.522543 26373 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22b70b0546574e2f8a4a3a3962a82b45 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 22b70b0546574e2f8a4a3a3962a82b45. Latest consensus state: current_term: 1 leader_uuid: "22b70b0546574e2f8a4a3a3962a82b45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22b70b0546574e2f8a4a3a3962a82b45" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 44799 } } }
I20260926 07:30:06.522673 26373 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22b70b0546574e2f8a4a3a3962a82b45 [sys.catalog]: This master's current role is: LEADER
I20260926 07:30:06.524365 26382 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:30:06.525921 26374 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22b70b0546574e2f8a4a3a3962a82b45 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "22b70b0546574e2f8a4a3a3962a82b45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22b70b0546574e2f8a4a3a3962a82b45" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 44799 } } }
I20260926 07:30:06.526145 26374 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22b70b0546574e2f8a4a3a3962a82b45 [sys.catalog]: This master's current role is: LEADER
I20260926 07:30:06.528681 26382 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:30:06.535238 26382 catalog_manager.cc:1347] Generated new cluster ID: affe06b73a314985850a77183c6ea1cd
I20260926 07:30:06.535382 26382 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:30:06.551503 26382 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:30:06.552697 26382 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:30:06.564903 26382 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 22b70b0546574e2f8a4a3a3962a82b45: Generated new TSK 0
I20260926 07:30:06.565829 26382 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.53.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.53.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.53.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.193.keytab.
Entry for principal kudu/127.22.53.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.193.keytab.
I20260926 07:30:06.606150 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407783529717-22743-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407783529717-22743-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407783529717-22743-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407783529717-22743-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.53.193:0
--local_ip_for_outbound_sockets=127.22.53.193
--webserver_interface=127.22.53.193
--webserver_port=0
--tserver_master_addrs=127.22.53.254:44799
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.193.keytab
--principal=kudu/127.22.53.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:37411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:30:06.739060 26394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:30:06.739382 26394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:30:06.739431 26394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:30:06.744240 26394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:30:06.744333 26394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:30:06.744403 26394 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.53.193
I20260926 07:30:06.750536 26394 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:37411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407783529717-22743-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407783529717-22743-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.193.keytab
--principal=kudu/127.22.53.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.53.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407783529717-22743-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.53.193
--webserver_port=0
--tserver_master_addrs=127.22.53.254:44799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26394
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.53.193
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407783529717-22743-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:30:06.751804 26394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:30:06.753070 26394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:30:06.764807 26400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:06.764815 26402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:06.764816 26399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:06.765281 26394 server_base.cc:1034] running on GCE node
I20260926 07:30:06.765918 26394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:30:06.767062 26394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:30:06.768352 26394 hybrid_clock.cc:648] HybridClock initialized: now 1790407806768324 us; error 48 us; skew 500 ppm
Sep 26 07:30:06 dist-test-slave-wdjc krb5kdc[26287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407806, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:30:06.772007 26394 init.cc:377] Logged in from keytab as kudu/127.22.53.193@KRBTEST.COM (short username kudu)
I20260926 07:30:06.773196 26394 webserver.cc:460] Webserver started at http://127.22.53.193:41853/ using document root <none> and password file <none>
I20260926 07:30:06.773797 26394 fs_manager.cc:359] Metadata directory not provided
I20260926 07:30:06.773869 26394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:30:06.774083 26394 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:30:06.776446 26394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407783529717-22743-0/minicluster-data/ts-0/data/instance:
uuid: "e8ec792c643d4ef4ab3ec1ec3adee9e4"
format_stamp: "Formatted at 2026-09-26 07:30:06 on dist-test-slave-wdjc"
server_key: "90e5975010cba686e5f36103a67709f2"
server_key_iv: "bbfc931b7c9dd8289f7743b07d027f0c"
server_key_version: "encryptionkey@0"
I20260926 07:30:06.777145 26394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407783529717-22743-0/minicluster-data/ts-0/wal/instance:
uuid: "e8ec792c643d4ef4ab3ec1ec3adee9e4"
format_stamp: "Formatted at 2026-09-26 07:30:06 on dist-test-slave-wdjc"
server_key: "90e5975010cba686e5f36103a67709f2"
server_key_iv: "bbfc931b7c9dd8289f7743b07d027f0c"
server_key_version: "encryptionkey@0"
I20260926 07:30:06.782087 26394 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260926 07:30:06.785645 26409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:06.787351 26394 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20260926 07:30:06.787590 26394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407783529717-22743-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407783529717-22743-0/minicluster-data/ts-0/wal
uuid: "e8ec792c643d4ef4ab3ec1ec3adee9e4"
format_stamp: "Formatted at 2026-09-26 07:30:06 on dist-test-slave-wdjc"
server_key: "90e5975010cba686e5f36103a67709f2"
server_key_iv: "bbfc931b7c9dd8289f7743b07d027f0c"
server_key_version: "encryptionkey@0"
I20260926 07:30:06.787771 26394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407783529717-22743-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407783529717-22743-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407783529717-22743-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:30:06.841388 26394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:30:06.845463 26394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:30:06.845729 26394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:30:06.846491 26394 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:30:06.847559 26394 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:30:06.847651 26394 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:06.847730 26394 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:30:06.847777 26394 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:06.858946 26394 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.193:37547
I20260926 07:30:06.858989 26522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.193:37547 every 8 connection(s)
I20260926 07:30:06.860065 26394 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407783529717-22743-0/minicluster-data/ts-0/data/info.pb
I20260926 07:30:06.866775 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 26394
I20260926 07:30:06.866956 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407783529717-22743-0/minicluster-data/ts-0/wal/instance
I20260926 07:30:06.867296 22743 external_mini_cluster.cc:1428] Setting key bacfbd7a3ae18caccfd94b298c5d23d8
Sep 26 07:30:06 dist-test-slave-wdjc krb5kdc[26287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407806, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.193@KRBTEST.COM for kudu/127.22.53.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.22.53.194@KRBTEST.COM; defaulting to no policy
I20260926 07:30:06.877898 26371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:06.862340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.193:39671 (local address 127.22.53.254:44799)
0926 07:30:06.862496 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:06.862501 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:06.864001 (+ 1500us) server_negotiation.cc:407] Connection header received
0926 07:30:06.865630 (+ 1629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:06.865644 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:06.865729 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:06.865857 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:06.867534 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:06.868227 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:06.869700 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:06.869948 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:06.870080 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:06.873363 (+ 3283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:06.873368 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:06.873371 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:06.873403 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:06.875146 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:06.875875 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:06.875879 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:06.875881 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:06.875970 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:06.876412 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:06.876416 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:06.876418 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:06.876731 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:06.876858 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:06.877530 (+ 672us) server_negotiation.cc:299] Negotiation successful
0926 07:30:06.877694 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 07:30:06.878934 26525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:06.862765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:44799 (local address 127.22.53.193:39671)
0926 07:30:06.863592 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:06.863799 (+ 207us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:06.865279 (+ 1480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:06.866114 (+ 835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:06.866122 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:06.866594 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:06.867291 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:06.867308 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:06.868391 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:06.868395 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:06.869544 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:06.869556 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:06.870153 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:06.870157 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:06.870354 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:06.871095 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:06.871125 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:06.873102 (+ 1977us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:30:06.875335 (+ 2233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:06.875343 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:06.875377 (+ 34us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:06.875724 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:06.876086 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:06.876090 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:06.876092 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:06.876282 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:06.876870 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:30:06.876882 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:30:06.877214 (+ 332us) client_negotiation.cc:769] Sending connection context
0926 07:30:06.877467 (+ 253us) client_negotiation.cc:240] Negotiation successful
0926 07:30:06.877870 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":141,"threads_started":1}
I20260926 07:30:06.881006 26523 heartbeater.cc:347] Connected to a master server at 127.22.53.254:44799
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.53.194@KRBTEST.COM" created.
I20260926 07:30:06.881532 26523 heartbeater.cc:464] Registering TS with master...
I20260926 07:30:06.882370 26523 heartbeater.cc:511] Master 127.22.53.254:44799 requested a full tablet report, sending...
I20260926 07:30:06.884335 26330 ts_manager.cc:194] Registered new tserver with Master: e8ec792c643d4ef4ab3ec1ec3adee9e4 (127.22.53.193:37547)
I20260926 07:30:06.886137 26330 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.53.193@KRBTEST.COM'} at 127.22.53.193:39671
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.53.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.194.keytab.
Entry for principal kudu/127.22.53.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.194.keytab.
I20260926 07:30:06.900429 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407783529717-22743-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407783529717-22743-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407783529717-22743-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407783529717-22743-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.53.194:0
--local_ip_for_outbound_sockets=127.22.53.194
--webserver_interface=127.22.53.194
--webserver_port=0
--tserver_master_addrs=127.22.53.254:44799
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.194.keytab
--principal=kudu/127.22.53.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:37411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:30:07.029039 26528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:30:07.029381 26528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:30:07.029428 26528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:30:07.034096 26528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:30:07.034243 26528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:30:07.034363 26528 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.53.194
I20260926 07:30:07.041112 26528 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:37411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407783529717-22743-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407783529717-22743-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.194.keytab
--principal=kudu/127.22.53.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.53.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407783529717-22743-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.53.194
--webserver_port=0
--tserver_master_addrs=127.22.53.254:44799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26528
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.53.194
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407783529717-22743-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:30:07.042376 26528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:30:07.043512 26528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 07:30:07.055114 26528 server_base.cc:1034] running on GCE node
W20260926 07:30:07.055065 26533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:07.055126 26534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:07.055065 26536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:07.055804 26528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:30:07.056875 26528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:30:07.058074 26528 hybrid_clock.cc:648] HybridClock initialized: now 1790407807058059 us; error 59 us; skew 500 ppm
Sep 26 07:30:07 dist-test-slave-wdjc krb5kdc[26287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407807, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:30:07.061436 26528 init.cc:377] Logged in from keytab as kudu/127.22.53.194@KRBTEST.COM (short username kudu)
I20260926 07:30:07.062549 26528 webserver.cc:460] Webserver started at http://127.22.53.194:40193/ using document root <none> and password file <none>
I20260926 07:30:07.063206 26528 fs_manager.cc:359] Metadata directory not provided
I20260926 07:30:07.063308 26528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:30:07.063565 26528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:30:07.066039 26528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407783529717-22743-0/minicluster-data/ts-1/data/instance:
uuid: "42fbe15af7fa4b89b21fccd7eeaac938"
format_stamp: "Formatted at 2026-09-26 07:30:07 on dist-test-slave-wdjc"
server_key: "d03136c50d40bedee5b221b268e10260"
server_key_iv: "ab205cdbdedde399a05e3c923b513c4d"
server_key_version: "encryptionkey@0"
I20260926 07:30:07.066918 26528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407783529717-22743-0/minicluster-data/ts-1/wal/instance:
uuid: "42fbe15af7fa4b89b21fccd7eeaac938"
format_stamp: "Formatted at 2026-09-26 07:30:07 on dist-test-slave-wdjc"
server_key: "d03136c50d40bedee5b221b268e10260"
server_key_iv: "ab205cdbdedde399a05e3c923b513c4d"
server_key_version: "encryptionkey@0"
I20260926 07:30:07.071796 26528 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 07:30:07.074932 26543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:07.076491 26528 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 07:30:07.076746 26528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407783529717-22743-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407783529717-22743-0/minicluster-data/ts-1/wal
uuid: "42fbe15af7fa4b89b21fccd7eeaac938"
format_stamp: "Formatted at 2026-09-26 07:30:07 on dist-test-slave-wdjc"
server_key: "d03136c50d40bedee5b221b268e10260"
server_key_iv: "ab205cdbdedde399a05e3c923b513c4d"
server_key_version: "encryptionkey@0"
I20260926 07:30:07.076897 26528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407783529717-22743-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407783529717-22743-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407783529717-22743-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:30:07.091445 26528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:30:07.095942 26528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:30:07.096215 26528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:30:07.097007 26528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:30:07.098196 26528 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:30:07.098282 26528 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:07.098364 26528 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:30:07.098413 26528 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:07.109182 26528 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.194:34525
I20260926 07:30:07.109221 26656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.194:34525 every 8 connection(s)
I20260926 07:30:07.110236 26528 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407783529717-22743-0/minicluster-data/ts-1/data/info.pb
I20260926 07:30:07.118963 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 26528
I20260926 07:30:07.119174 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407783529717-22743-0/minicluster-data/ts-1/wal/instance
I20260926 07:30:07.119534 22743 external_mini_cluster.cc:1428] Setting key fa1b1cef276a94f4cf980b9842cb284a
Sep 26 07:30:07 dist-test-slave-wdjc krb5kdc[26287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407807, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.194@KRBTEST.COM for kudu/127.22.53.254@KRBTEST.COM
I20260926 07:30:07.129046 26371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:07.112476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.194:45323 (local address 127.22.53.254:44799)
0926 07:30:07.112699 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:07.112703 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:07.113756 (+ 1053us) server_negotiation.cc:407] Connection header received
0926 07:30:07.115421 (+ 1665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:07.115424 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:07.115495 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:07.115632 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:07.117063 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:07.117639 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:07.119045 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:07.119370 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:07.119501 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:07.123210 (+ 3709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:07.123218 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:07.123225 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:07.123267 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:07.125438 (+ 2171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:07.126453 (+ 1015us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:07.126459 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:07.126460 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:07.126637 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:07.127337 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:07.127344 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:07.127352 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:07.127612 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:07.127725 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:07.128739 (+ 1014us) server_negotiation.cc:299] Negotiation successful
0926 07:30:07.128878 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260926 07:30:07.130451 26659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:07.112968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:44799 (local address 127.22.53.194:45323)
0926 07:30:07.113542 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:07.113603 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:07.115124 (+ 1521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:07.115848 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:07.115857 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:07.116286 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:07.116869 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:07.116884 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:07.117819 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:07.117825 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:07.118880 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:07.118893 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:07.119564 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:07.119569 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:07.119801 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:07.120662 (+ 861us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:07.120689 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:07.122959 (+ 2270us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:30:07.125624 (+ 2665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:07.125636 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:07.125662 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:07.126169 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:07.126804 (+ 635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:07.126811 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:07.126817 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:07.127107 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:07.127769 (+ 662us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:30:07.127781 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:30:07.128252 (+ 471us) client_negotiation.cc:769] Sending connection context
0926 07:30:07.128591 (+ 339us) client_negotiation.cc:240] Negotiation successful
0926 07:30:07.129085 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":150,"threads_started":1}
I20260926 07:30:07.132264 26657 heartbeater.cc:347] Connected to a master server at 127.22.53.254:44799
WARNING: no policy specified for kudu/127.22.53.195@KRBTEST.COM; defaulting to no policy
I20260926 07:30:07.132673 26657 heartbeater.cc:464] Registering TS with master...
I20260926 07:30:07.133298 26657 heartbeater.cc:511] Master 127.22.53.254:44799 requested a full tablet report, sending...
I20260926 07:30:07.134652 26330 ts_manager.cc:194] Registered new tserver with Master: 42fbe15af7fa4b89b21fccd7eeaac938 (127.22.53.194:34525)
I20260926 07:30:07.135336 26330 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.53.194@KRBTEST.COM'} at 127.22.53.194:45323
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.53.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.53.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.195.keytab.
Entry for principal kudu/127.22.53.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.195.keytab.
I20260926 07:30:07.155929 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407783529717-22743-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407783529717-22743-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407783529717-22743-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407783529717-22743-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.53.195:0
--local_ip_for_outbound_sockets=127.22.53.195
--webserver_interface=127.22.53.195
--webserver_port=0
--tserver_master_addrs=127.22.53.254:44799
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.195.keytab
--principal=kudu/127.22.53.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:37411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:30:07.281584 26662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:30:07.281962 26662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:30:07.282012 26662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:30:07.286538 26662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:30:07.286624 26662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:30:07.286691 26662 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.53.195
I20260926 07:30:07.292802 26662 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:37411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407783529717-22743-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407783529717-22743-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.195.keytab
--principal=kudu/127.22.53.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.53.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407783529717-22743-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.53.195
--webserver_port=0
--tserver_master_addrs=127.22.53.254:44799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26662
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.53.195
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407783529717-22743-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:30:07.294305 26662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:30:07.295431 26662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 07:30:07.306771 26662 server_base.cc:1034] running on GCE node
W20260926 07:30:07.306694 26668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:07.306703 26667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:07.306699 26670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:07.307507 26662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:30:07.308310 26662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:30:07.309530 26662 hybrid_clock.cc:648] HybridClock initialized: now 1790407807309497 us; error 64 us; skew 500 ppm
Sep 26 07:30:07 dist-test-slave-wdjc krb5kdc[26287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407807, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:30:07.313760 26662 init.cc:377] Logged in from keytab as kudu/127.22.53.195@KRBTEST.COM (short username kudu)
I20260926 07:30:07.315021 26662 webserver.cc:460] Webserver started at http://127.22.53.195:44417/ using document root <none> and password file <none>
I20260926 07:30:07.315703 26662 fs_manager.cc:359] Metadata directory not provided
I20260926 07:30:07.315816 26662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:30:07.316220 26662 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:30:07.318708 26662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407783529717-22743-0/minicluster-data/ts-2/data/instance:
uuid: "4043b01c4caa494d9676889bc4dd5058"
format_stamp: "Formatted at 2026-09-26 07:30:07 on dist-test-slave-wdjc"
server_key: "d4b02598033dd27f7e67c7ae08f6d128"
server_key_iv: "095a5b038b61cb23b6b4b28906bf4c99"
server_key_version: "encryptionkey@0"
I20260926 07:30:07.319435 26662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407783529717-22743-0/minicluster-data/ts-2/wal/instance:
uuid: "4043b01c4caa494d9676889bc4dd5058"
format_stamp: "Formatted at 2026-09-26 07:30:07 on dist-test-slave-wdjc"
server_key: "d4b02598033dd27f7e67c7ae08f6d128"
server_key_iv: "095a5b038b61cb23b6b4b28906bf4c99"
server_key_version: "encryptionkey@0"
I20260926 07:30:07.324242 26662 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260926 07:30:07.327643 26677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:07.329321 26662 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:30:07.329517 26662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407783529717-22743-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407783529717-22743-0/minicluster-data/ts-2/wal
uuid: "4043b01c4caa494d9676889bc4dd5058"
format_stamp: "Formatted at 2026-09-26 07:30:07 on dist-test-slave-wdjc"
server_key: "d4b02598033dd27f7e67c7ae08f6d128"
server_key_iv: "095a5b038b61cb23b6b4b28906bf4c99"
server_key_version: "encryptionkey@0"
I20260926 07:30:07.329684 26662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407783529717-22743-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407783529717-22743-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407783529717-22743-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:30:07.346248 26662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:30:07.350795 26662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:30:07.351060 26662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:30:07.351761 26662 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:30:07.352803 26662 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:30:07.352877 26662 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:07.352965 26662 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:30:07.353018 26662 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:07.365680 26662 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.195:37401
I20260926 07:30:07.365724 26790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.195:37401 every 8 connection(s)
I20260926 07:30:07.366689 26662 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407783529717-22743-0/minicluster-data/ts-2/data/info.pb
I20260926 07:30:07.374588 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 26662
I20260926 07:30:07.374922 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407783529717-22743-0/minicluster-data/ts-2/wal/instance
I20260926 07:30:07.375361 22743 external_mini_cluster.cc:1428] Setting key fe9a0fb22917f855544ded8422dcfb02
Sep 26 07:30:07 dist-test-slave-wdjc krb5kdc[26287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407807, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.195@KRBTEST.COM for kudu/127.22.53.254@KRBTEST.COM
I20260926 07:30:07.384126 26371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:07.368941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.195:46627 (local address 127.22.53.254:44799)
0926 07:30:07.369141 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:07.369146 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:07.370168 (+ 1022us) server_negotiation.cc:407] Connection header received
0926 07:30:07.371718 (+ 1550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:07.371721 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:07.371785 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:07.371880 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:07.373791 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:07.374473 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:07.376415 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:07.376738 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:07.376866 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:07.379783 (+ 2917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:07.379786 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:07.379789 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:07.379817 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:07.381569 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:07.382346 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:07.382350 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:07.382352 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:07.382449 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:07.382842 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:07.382847 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:07.382848 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:07.383071 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:07.383187 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:07.383781 (+ 594us) server_negotiation.cc:299] Negotiation successful
0926 07:30:07.383936 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 07:30:07.385162 26793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:07.369384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:44799 (local address 127.22.53.195:46627)
0926 07:30:07.369982 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:07.370034 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:07.371460 (+ 1426us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:07.372357 (+ 897us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:07.372367 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:07.372983 (+ 616us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:07.373555 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:07.373571 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:07.374827 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:07.374833 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:07.376208 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:07.376223 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:07.376928 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:07.376932 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:07.377153 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:07.377860 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:07.377882 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:07.379537 (+ 1655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:30:07.381740 (+ 2203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:07.381748 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:07.381772 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:07.382216 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:07.382560 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:07.382563 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:07.382565 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:07.382735 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:07.383185 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:30:07.383194 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:30:07.383519 (+ 325us) client_negotiation.cc:769] Sending connection context
0926 07:30:07.383743 (+ 224us) client_negotiation.cc:240] Negotiation successful
0926 07:30:07.384150 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":148,"threads_started":1}
I20260926 07:30:07.386620 26791 heartbeater.cc:347] Connected to a master server at 127.22.53.254:44799
I20260926 07:30:07.386929 26791 heartbeater.cc:464] Registering TS with master...
I20260926 07:30:07.387532 26791 heartbeater.cc:511] Master 127.22.53.254:44799 requested a full tablet report, sending...
I20260926 07:30:07.388926 26330 ts_manager.cc:194] Registered new tserver with Master: 4043b01c4caa494d9676889bc4dd5058 (127.22.53.195:37401)
I20260926 07:30:07.389803 26330 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.53.195@KRBTEST.COM'} at 127.22.53.195:46627
I20260926 07:30:07.390712 22743 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 07:30:07 dist-test-slave-wdjc krb5kdc[26287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407807, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 07:30:07 dist-test-slave-wdjc krb5kdc[26287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407807, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.53.193@KRBTEST.COM
I20260926 07:30:07.422994 26798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:07.410790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57686 (local address 127.22.53.193:37547)
0926 07:30:07.411170 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:07.411181 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:07.411213 (+ 32us) server_negotiation.cc:407] Connection header received
0926 07:30:07.411428 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:07.411436 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:07.411675 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:07.411848 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:07.412748 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:07.413571 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:07.414494 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:07.414923 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:07.415074 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:07.418107 (+ 3033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:07.418120 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:07.418137 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:07.418170 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:07.420249 (+ 2079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:07.420864 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:07.420871 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:07.420872 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:07.420971 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:07.421633 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:07.421639 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:07.421644 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:07.421973 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:07.422198 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:07.422534 (+ 336us) server_negotiation.cc:299] Negotiation successful
0926 07:30:07.422747 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":76,"threads_started":1}
Sep 26 07:30:07 dist-test-slave-wdjc krb5kdc[26287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407807, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.53.254@KRBTEST.COM
I20260926 07:30:07.437440 26371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:07.426564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60626 (local address 127.22.53.254:44799)
0926 07:30:07.426712 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:07.426717 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:07.426886 (+ 169us) server_negotiation.cc:407] Connection header received
0926 07:30:07.427061 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:07.427096 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:07.427145 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:07.427239 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:07.427939 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:07.428481 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:07.429379 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:07.429609 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:07.429728 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:07.432889 (+ 3161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:07.432893 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:07.432895 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:07.432933 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:07.434982 (+ 2049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:07.435732 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:07.435747 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:07.435749 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:07.435880 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:07.436289 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:07.436292 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:07.436293 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:07.436659 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:07.436784 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:07.437109 (+ 325us) server_negotiation.cc:299] Negotiation successful
0926 07:30:07.437230 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260926 07:30:07.438012 26331 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60626
I20260926 07:30:07.438987 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 26394
I20260926 07:30:07.445170 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 26528
I20260926 07:30:07.451160 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 26662
I20260926 07:30:07.457291 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 26300
2026-09-26T07:30:07Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2438 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407783529717-22743-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:30:07 dist-test-slave-wdjc krb5kdc[26803](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:30:07 dist-test-slave-wdjc krb5kdc[26803](info): set up 2 sockets
Sep 26 07:30:07 dist-test-slave-wdjc krb5kdc[26803](info): commencing operation
krb5kdc: starting...
W20260926 07:30:08.521869 22743 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407783529717-22743-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407783529717-22743-0/krb5kdc/test-user.keytab.
Sep 26 07:30:08 dist-test-slave-wdjc krb5kdc[26803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407808, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:30:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:30:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.53.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.53.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.53.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab.
Entry for principal kudu/127.22.53.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab.
I20260926 07:30:08.660084 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407783529717-22743-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407783529717-22743-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407783529717-22743-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.53.254:38209
--webserver_interface=127.22.53.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab
--principal=kudu/127.22.53.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:33865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.53.254:38209
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:30:08.786118 26816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:30:08.786473 26816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:30:08.786521 26816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:30:08.791224 26816 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:30:08.791324 26816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:30:08.791345 26816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:30:08.791364 26816 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:30:08.791380 26816 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:30:08.798146 26816 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:33865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407783529717-22743-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407783529717-22743-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.53.254:38209
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab
--principal=kudu/127.22.53.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.53.254:38209
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.53.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26816
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407783529717-22743-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:30:08.799470 26816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:30:08.800742 26816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:30:08.811249 26822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:08.811344 26824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:08.811276 26821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:08.811950 26816 server_base.cc:1034] running on GCE node
I20260926 07:30:08.812740 26816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:30:08.814073 26816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:30:08.815403 26816 hybrid_clock.cc:648] HybridClock initialized: now 1790407808815370 us; error 42 us; skew 500 ppm
Sep 26 07:30:08 dist-test-slave-wdjc krb5kdc[26803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407808, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:30:08.819638 26816 init.cc:377] Logged in from keytab as kudu/127.22.53.254@KRBTEST.COM (short username kudu)
I20260926 07:30:08.820678 26816 fs_manager.cc:359] Metadata directory not provided
I20260926 07:30:08.820802 26816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:30:08.821086 26816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:30:08.823733 26816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407783529717-22743-0/minicluster-data/master-0/data/instance:
uuid: "e65556604b494921bfba02647d624667"
format_stamp: "Formatted at 2026-09-26 07:30:08 on dist-test-slave-wdjc"
server_key: "9552662f1bdd4af61638c47e612185f6"
server_key_iv: "4a74bf161390ea5a17f22d735cd3da2f"
server_key_version: "encryptionkey@0"
I20260926 07:30:08.824435 26816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407783529717-22743-0/minicluster-data/master-0/wal/instance:
uuid: "e65556604b494921bfba02647d624667"
format_stamp: "Formatted at 2026-09-26 07:30:08 on dist-test-slave-wdjc"
server_key: "9552662f1bdd4af61638c47e612185f6"
server_key_iv: "4a74bf161390ea5a17f22d735cd3da2f"
server_key_version: "encryptionkey@0"
I20260926 07:30:08.829188 26816 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 07:30:08.832285 26830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:08.834007 26816 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:30:08.834218 26816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407783529717-22743-0/minicluster-data/master-0/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407783529717-22743-0/minicluster-data/master-0/wal
uuid: "e65556604b494921bfba02647d624667"
format_stamp: "Formatted at 2026-09-26 07:30:08 on dist-test-slave-wdjc"
server_key: "9552662f1bdd4af61638c47e612185f6"
server_key_iv: "4a74bf161390ea5a17f22d735cd3da2f"
server_key_version: "encryptionkey@0"
I20260926 07:30:08.834378 26816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407783529717-22743-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407783529717-22743-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407783529717-22743-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:30:08.854113 26816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:30:08.858036 26816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:30:08.858310 26816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:30:08.867314 26816 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.254:38209
I20260926 07:30:08.867698 26882 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.254:38209 every 8 connection(s)
I20260926 07:30:08.868165 26816 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
I20260926 07:30:08.868942 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 26816
I20260926 07:30:08.869099 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407783529717-22743-0/minicluster-data/master-0/wal/instance
I20260926 07:30:08.869408 22743 external_mini_cluster.cc:1428] Setting key bf784c0531f760dc3c12ee544b0bafdc
I20260926 07:30:08.872352 26883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:30:08.880174 26883 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e65556604b494921bfba02647d624667: Bootstrap starting.
Sep 26 07:30:08 dist-test-slave-wdjc krb5kdc[26803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407808, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.53.254@KRBTEST.COM
I20260926 07:30:08.883038 26883 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e65556604b494921bfba02647d624667: Neither blocks nor log segments found. Creating new log.
I20260926 07:30:08.884220 26883 log.cc:824] T 00000000000000000000000000000000 P e65556604b494921bfba02647d624667: Log is configured to *not* fsync() on all Append() calls
I20260926 07:30:08.887154 26883 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e65556604b494921bfba02647d624667: No bootstrap required, opened a new log
I20260926 07:30:08.888345 26886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:08.871774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38376 (local address 127.22.53.254:38209)
0926 07:30:08.872337 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:08.872366 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:08.872407 (+ 41us) server_negotiation.cc:407] Connection header received
0926 07:30:08.873310 (+ 903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:08.873330 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:08.873714 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:08.874051 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:08.875101 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:08.876212 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:08.877480 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:08.878024 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:08.878170 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:08.882121 (+ 3951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:08.882128 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:08.882145 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:08.882189 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:08.884434 (+ 2245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:08.885226 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:08.885231 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:08.885237 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:08.885436 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:08.885776 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:08.885779 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:08.885781 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:08.886346 (+ 565us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:08.886571 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:08.886922 (+ 351us) server_negotiation.cc:299] Negotiation successful
0926 07:30:08.887304 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":242,"threads_started":1}
I20260926 07:30:08.891746 26883 raft_consensus.cc:348] T 00000000000000000000000000000000 P e65556604b494921bfba02647d624667 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e65556604b494921bfba02647d624667" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 38209 } }
I20260926 07:30:08.892024 26883 raft_consensus.cc:374] T 00000000000000000000000000000000 P e65556604b494921bfba02647d624667 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:30:08.892061 26883 raft_consensus.cc:729] T 00000000000000000000000000000000 P e65556604b494921bfba02647d624667 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e65556604b494921bfba02647d624667, State: Initialized, Role: FOLLOWER
I20260926 07:30:08.892582 26883 consensus_queue.cc:260] T 00000000000000000000000000000000 P e65556604b494921bfba02647d624667 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e65556604b494921bfba02647d624667" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 38209 } }
I20260926 07:30:08.892741 26883 raft_consensus.cc:388] T 00000000000000000000000000000000 P e65556604b494921bfba02647d624667 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:30:08.892797 26883 raft_consensus.cc:482] T 00000000000000000000000000000000 P e65556604b494921bfba02647d624667 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:30:08.892877 26883 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e65556604b494921bfba02647d624667 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:30:08.894018 26883 raft_consensus.cc:504] T 00000000000000000000000000000000 P e65556604b494921bfba02647d624667 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e65556604b494921bfba02647d624667" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 38209 } }
I20260926 07:30:08.894469 26883 leader_election.cc:302] T 00000000000000000000000000000000 P e65556604b494921bfba02647d624667 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e65556604b494921bfba02647d624667; no voters:
I20260926 07:30:08.894953 26883 leader_election.cc:288] T 00000000000000000000000000000000 P e65556604b494921bfba02647d624667 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:30:08.895205 26888 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e65556604b494921bfba02647d624667 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:30:08.895543 26888 raft_consensus.cc:686] T 00000000000000000000000000000000 P e65556604b494921bfba02647d624667 [term 1 LEADER]: Becoming Leader. State: Replica: e65556604b494921bfba02647d624667, State: Running, Role: LEADER
I20260926 07:30:08.896026 26888 consensus_queue.cc:237] T 00000000000000000000000000000000 P e65556604b494921bfba02647d624667 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e65556604b494921bfba02647d624667" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 38209 } }
I20260926 07:30:08.896710 26883 sys_catalog.cc:564] T 00000000000000000000000000000000 P e65556604b494921bfba02647d624667 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:30:08.898002 26890 sys_catalog.cc:455] T 00000000000000000000000000000000 P e65556604b494921bfba02647d624667 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e65556604b494921bfba02647d624667. Latest consensus state: current_term: 1 leader_uuid: "e65556604b494921bfba02647d624667" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e65556604b494921bfba02647d624667" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 38209 } } }
I20260926 07:30:08.898175 26890 sys_catalog.cc:458] T 00000000000000000000000000000000 P e65556604b494921bfba02647d624667 [sys.catalog]: This master's current role is: LEADER
I20260926 07:30:08.898533 26889 sys_catalog.cc:455] T 00000000000000000000000000000000 P e65556604b494921bfba02647d624667 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e65556604b494921bfba02647d624667" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e65556604b494921bfba02647d624667" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 38209 } } }
I20260926 07:30:08.898641 26889 sys_catalog.cc:458] T 00000000000000000000000000000000 P e65556604b494921bfba02647d624667 [sys.catalog]: This master's current role is: LEADER
I20260926 07:30:08.898746 26897 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:30:08.903076 26897 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:30:08.912081 26897 catalog_manager.cc:1347] Generated new cluster ID: aeac01fe2d6a48539222a5c7734bc471
I20260926 07:30:08.912220 26897 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:30:08.948578 26897 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:30:08.950116 26897 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:30:08.957508 26897 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e65556604b494921bfba02647d624667: Generated new TSK 0
I20260926 07:30:08.958370 26897 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.53.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.53.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.53.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.193.keytab.
Entry for principal kudu/127.22.53.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.193.keytab.
I20260926 07:30:08.997465 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407783529717-22743-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407783529717-22743-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407783529717-22743-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407783529717-22743-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.53.193:0
--local_ip_for_outbound_sockets=127.22.53.193
--webserver_interface=127.22.53.193
--webserver_port=0
--tserver_master_addrs=127.22.53.254:38209
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.193.keytab
--principal=kudu/127.22.53.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:33865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:30:09.130786 26909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:30:09.131112 26909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:30:09.131163 26909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:30:09.135545 26909 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:30:09.135644 26909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:30:09.135711 26909 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.53.193
I20260926 07:30:09.141924 26909 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:33865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407783529717-22743-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407783529717-22743-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.193.keytab
--principal=kudu/127.22.53.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.53.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407783529717-22743-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.53.193
--webserver_port=0
--tserver_master_addrs=127.22.53.254:38209
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26909
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.53.193
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407783529717-22743-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:30:09.143572 26909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:30:09.145260 26909 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:30:09.157920 26917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:09.157985 26915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:09.157920 26914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:09.158885 26909 server_base.cc:1034] running on GCE node
I20260926 07:30:09.159384 26909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:30:09.160121 26909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:30:09.163522 26909 hybrid_clock.cc:648] HybridClock initialized: now 1790407809163396 us; error 117 us; skew 500 ppm
Sep 26 07:30:09 dist-test-slave-wdjc krb5kdc[26803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407809, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:30:09.167805 26909 init.cc:377] Logged in from keytab as kudu/127.22.53.193@KRBTEST.COM (short username kudu)
I20260926 07:30:09.168869 26909 fs_manager.cc:359] Metadata directory not provided
I20260926 07:30:09.168954 26909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:30:09.169206 26909 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:30:09.172113 26909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407783529717-22743-0/minicluster-data/ts-0/data/instance:
uuid: "49a79ddb38674ef9b143c90eaedd8c38"
format_stamp: "Formatted at 2026-09-26 07:30:09 on dist-test-slave-wdjc"
server_key: "e4699d5fe9a1fc8521f01397bec82cd6"
server_key_iv: "3b02a19ed6ba965b8635d4b57481cc45"
server_key_version: "encryptionkey@0"
I20260926 07:30:09.172854 26909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407783529717-22743-0/minicluster-data/ts-0/wal/instance:
uuid: "49a79ddb38674ef9b143c90eaedd8c38"
format_stamp: "Formatted at 2026-09-26 07:30:09 on dist-test-slave-wdjc"
server_key: "e4699d5fe9a1fc8521f01397bec82cd6"
server_key_iv: "3b02a19ed6ba965b8635d4b57481cc45"
server_key_version: "encryptionkey@0"
I20260926 07:30:09.177855 26909 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 07:30:09.181208 26923 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:09.182925 26909 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 07:30:09.183144 26909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407783529717-22743-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407783529717-22743-0/minicluster-data/ts-0/wal
uuid: "49a79ddb38674ef9b143c90eaedd8c38"
format_stamp: "Formatted at 2026-09-26 07:30:09 on dist-test-slave-wdjc"
server_key: "e4699d5fe9a1fc8521f01397bec82cd6"
server_key_iv: "3b02a19ed6ba965b8635d4b57481cc45"
server_key_version: "encryptionkey@0"
I20260926 07:30:09.183306 26909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407783529717-22743-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407783529717-22743-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407783529717-22743-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:30:09.200977 26909 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:30:09.204993 26909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:30:09.205240 26909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:30:09.205943 26909 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:30:09.206923 26909 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:30:09.207003 26909 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:09.207077 26909 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:30:09.207123 26909 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:09.218005 26909 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.193:39007
I20260926 07:30:09.218042 27036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.193:39007 every 8 connection(s)
I20260926 07:30:09.218737 26909 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407783529717-22743-0/minicluster-data/ts-0/data/info.pb
I20260926 07:30:09.228042 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 26909
I20260926 07:30:09.228199 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407783529717-22743-0/minicluster-data/ts-0/wal/instance
I20260926 07:30:09.228613 22743 external_mini_cluster.cc:1428] Setting key ce43b775c38bd6af0bda39bd94e206fc
Sep 26 07:30:09 dist-test-slave-wdjc krb5kdc[26803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407809, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.193@KRBTEST.COM for kudu/127.22.53.254@KRBTEST.COM
I20260926 07:30:09.240080 26886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:09.221094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.193:49973 (local address 127.22.53.254:38209)
0926 07:30:09.221435 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:09.221440 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:09.222684 (+ 1244us) server_negotiation.cc:407] Connection header received
0926 07:30:09.223907 (+ 1223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:09.223911 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:09.223975 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:09.224069 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:09.225888 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:09.226637 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:09.228361 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:09.228826 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:09.228973 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:09.233374 (+ 4401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:09.233380 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:09.233385 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:09.233451 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:09.235929 (+ 2478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:09.236876 (+ 947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:09.236883 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:09.236884 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:09.237004 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:09.237929 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:09.237935 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:09.237939 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:09.238273 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:09.238415 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:09.239514 (+ 1099us) server_negotiation.cc:299] Negotiation successful
0926 07:30:09.239696 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
WARNING: no policy specified for kudu/127.22.53.194@KRBTEST.COM; defaulting to no policy
I20260926 07:30:09.240912 27039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:09.221375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:38209 (local address 127.22.53.193:49973)
0926 07:30:09.222395 (+ 1020us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:09.222450 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:09.223506 (+ 1056us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:09.224313 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:09.224322 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:09.224921 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:09.225555 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:09.225574 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:09.226883 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:09.226887 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:09.228057 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:09.228073 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:09.229042 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:09.229046 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:09.229347 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:09.230212 (+ 865us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:09.230233 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:09.233086 (+ 2853us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:30:09.236166 (+ 3080us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:09.236174 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:09.236192 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:09.236720 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:09.237225 (+ 505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:09.237230 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:09.237232 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:09.237621 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:09.238562 (+ 941us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:30:09.238571 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:30:09.239123 (+ 552us) client_negotiation.cc:769] Sending connection context
0926 07:30:09.239370 (+ 247us) client_negotiation.cc:240] Negotiation successful
0926 07:30:09.239786 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":137,"threads_started":1}
I20260926 07:30:09.242944 27037 heartbeater.cc:347] Connected to a master server at 127.22.53.254:38209
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.53.194@KRBTEST.COM" created.
I20260926 07:30:09.243341 27037 heartbeater.cc:464] Registering TS with master...
I20260926 07:30:09.243966 27037 heartbeater.cc:511] Master 127.22.53.254:38209 requested a full tablet report, sending...
I20260926 07:30:09.247249 26847 ts_manager.cc:194] Registered new tserver with Master: 49a79ddb38674ef9b143c90eaedd8c38 (127.22.53.193:39007)
I20260926 07:30:09.249495 26847 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.53.193@KRBTEST.COM'} at 127.22.53.193:49973
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.53.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.194.keytab.
Entry for principal kudu/127.22.53.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.194.keytab.
I20260926 07:30:09.269769 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407783529717-22743-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407783529717-22743-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407783529717-22743-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407783529717-22743-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.53.194:0
--local_ip_for_outbound_sockets=127.22.53.194
--webserver_interface=127.22.53.194
--webserver_port=0
--tserver_master_addrs=127.22.53.254:38209
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.194.keytab
--principal=kudu/127.22.53.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:33865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:30:09.397655 27042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:30:09.397962 27042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:30:09.398072 27042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:30:09.402725 27042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:30:09.402810 27042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:30:09.402880 27042 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.53.194
I20260926 07:30:09.409476 27042 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:33865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407783529717-22743-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407783529717-22743-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.194.keytab
--principal=kudu/127.22.53.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.53.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407783529717-22743-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.53.194
--webserver_port=0
--tserver_master_addrs=127.22.53.254:38209
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27042
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.53.194
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407783529717-22743-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:30:09.410699 27042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:30:09.411795 27042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:30:09.426424 27048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:09.426424 27047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:09.426424 27050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:09.426900 27042 server_base.cc:1034] running on GCE node
I20260926 07:30:09.427474 27042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:30:09.428287 27042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:30:09.429653 27042 hybrid_clock.cc:648] HybridClock initialized: now 1790407809429612 us; error 62 us; skew 500 ppm
Sep 26 07:30:09 dist-test-slave-wdjc krb5kdc[26803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407809, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:30:09.433516 27042 init.cc:377] Logged in from keytab as kudu/127.22.53.194@KRBTEST.COM (short username kudu)
I20260926 07:30:09.434417 27042 fs_manager.cc:359] Metadata directory not provided
I20260926 07:30:09.434492 27042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:30:09.434762 27042 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:30:09.437245 27042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407783529717-22743-0/minicluster-data/ts-1/data/instance:
uuid: "f62713828ed645318ecd9ab0e440ef72"
format_stamp: "Formatted at 2026-09-26 07:30:09 on dist-test-slave-wdjc"
server_key: "7c1083be2b3766cbe5f0f052844f23dc"
server_key_iv: "1c6d2edfca76e656da0b7263532371ac"
server_key_version: "encryptionkey@0"
I20260926 07:30:09.437914 27042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407783529717-22743-0/minicluster-data/ts-1/wal/instance:
uuid: "f62713828ed645318ecd9ab0e440ef72"
format_stamp: "Formatted at 2026-09-26 07:30:09 on dist-test-slave-wdjc"
server_key: "7c1083be2b3766cbe5f0f052844f23dc"
server_key_iv: "1c6d2edfca76e656da0b7263532371ac"
server_key_version: "encryptionkey@0"
I20260926 07:30:09.442488 27042 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 07:30:09.445487 27056 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:09.447124 27042 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:30:09.447263 27042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407783529717-22743-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407783529717-22743-0/minicluster-data/ts-1/wal
uuid: "f62713828ed645318ecd9ab0e440ef72"
format_stamp: "Formatted at 2026-09-26 07:30:09 on dist-test-slave-wdjc"
server_key: "7c1083be2b3766cbe5f0f052844f23dc"
server_key_iv: "1c6d2edfca76e656da0b7263532371ac"
server_key_version: "encryptionkey@0"
I20260926 07:30:09.447454 27042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407783529717-22743-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407783529717-22743-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407783529717-22743-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:30:09.470088 27042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:30:09.473704 27042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:30:09.473933 27042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:30:09.474611 27042 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:30:09.475751 27042 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:30:09.475811 27042 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:09.475853 27042 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:30:09.475870 27042 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:09.486284 27042 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.194:33315
I20260926 07:30:09.486299 27169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.194:33315 every 8 connection(s)
I20260926 07:30:09.487057 27042 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407783529717-22743-0/minicluster-data/ts-1/data/info.pb
I20260926 07:30:09.489214 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 27042
I20260926 07:30:09.489334 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407783529717-22743-0/minicluster-data/ts-1/wal/instance
I20260926 07:30:09.489645 22743 external_mini_cluster.cc:1428] Setting key 563aa994011d4ce1cfdada78ae6509f6
Sep 26 07:30:09 dist-test-slave-wdjc krb5kdc[26803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407809, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.194@KRBTEST.COM for kudu/127.22.53.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.22.53.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.53.195@KRBTEST.COM" created.
I20260926 07:30:09.506379 26886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:09.489155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.194:53463 (local address 127.22.53.254:38209)
0926 07:30:09.489339 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:09.489346 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:09.490389 (+ 1043us) server_negotiation.cc:407] Connection header received
0926 07:30:09.492163 (+ 1774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:09.492168 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:09.492252 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:09.492386 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:09.494093 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:09.494986 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:09.497043 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:09.497355 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:09.497511 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:09.501596 (+ 4085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:09.501600 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:09.501602 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:09.501635 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:09.503309 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:09.504117 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:09.504123 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:09.504125 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:09.504249 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:09.504838 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:09.504842 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:09.504845 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:09.505207 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:09.505334 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:09.506010 (+ 676us) server_negotiation.cc:299] Negotiation successful
0926 07:30:09.506221 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 07:30:09.507702 27172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:09.489624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:38209 (local address 127.22.53.194:53463)
0926 07:30:09.490199 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:09.490244 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:09.491778 (+ 1534us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:09.492683 (+ 905us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:09.492697 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:09.493290 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:09.493873 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:09.493890 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:09.495145 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:09.495152 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:09.496865 (+ 1713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:09.496884 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:09.497627 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:09.497636 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:09.497951 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:09.498702 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:09.498726 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:09.501338 (+ 2612us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:30:09.503502 (+ 2164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:09.503513 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:09.503536 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:09.503919 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:09.504392 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:09.504399 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:09.504404 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:09.504702 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:09.505354 (+ 652us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:30:09.505361 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:30:09.505691 (+ 330us) client_negotiation.cc:769] Sending connection context
0926 07:30:09.505890 (+ 199us) client_negotiation.cc:240] Negotiation successful
0926 07:30:09.506362 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":125,"threads_started":1}
I20260926 07:30:09.509255 27170 heartbeater.cc:347] Connected to a master server at 127.22.53.254:38209
I20260926 07:30:09.509598 27170 heartbeater.cc:464] Registering TS with master...
I20260926 07:30:09.510303 27170 heartbeater.cc:511] Master 127.22.53.254:38209 requested a full tablet report, sending...
I20260926 07:30:09.511866 26847 ts_manager.cc:194] Registered new tserver with Master: f62713828ed645318ecd9ab0e440ef72 (127.22.53.194:33315)
I20260926 07:30:09.512673 26847 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.53.194@KRBTEST.COM'} at 127.22.53.194:53463
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.53.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.195.keytab.
Entry for principal kudu/127.22.53.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.195.keytab.
I20260926 07:30:09.525452 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407783529717-22743-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407783529717-22743-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407783529717-22743-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407783529717-22743-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.53.195:0
--local_ip_for_outbound_sockets=127.22.53.195
--webserver_interface=127.22.53.195
--webserver_port=0
--tserver_master_addrs=127.22.53.254:38209
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.195.keytab
--principal=kudu/127.22.53.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:33865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:30:09.658160 27175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:30:09.658502 27175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:30:09.658552 27175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:30:09.663308 27175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:30:09.663394 27175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:30:09.663466 27175 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.53.195
I20260926 07:30:09.669584 27175 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:33865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407783529717-22743-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407783529717-22743-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.195.keytab
--principal=kudu/127.22.53.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.53.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407783529717-22743-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.53.195
--webserver_port=0
--tserver_master_addrs=127.22.53.254:38209
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27175
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.53.195
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407783529717-22743-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:30:09.670878 27175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:30:09.672079 27175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:30:09.683912 27180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:09.683912 27183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:09.683979 27175 server_base.cc:1034] running on GCE node
W20260926 07:30:09.683907 27181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:09.684845 27175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:30:09.685694 27175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:30:09.687062 27175 hybrid_clock.cc:648] HybridClock initialized: now 1790407809687032 us; error 45 us; skew 500 ppm
Sep 26 07:30:09 dist-test-slave-wdjc krb5kdc[26803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407809, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:30:09.690546 27175 init.cc:377] Logged in from keytab as kudu/127.22.53.195@KRBTEST.COM (short username kudu)
I20260926 07:30:09.691447 27175 fs_manager.cc:359] Metadata directory not provided
I20260926 07:30:09.691524 27175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:30:09.691787 27175 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:30:09.694296 27175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407783529717-22743-0/minicluster-data/ts-2/data/instance:
uuid: "432b20d79d01448586e187be5b2e0c66"
format_stamp: "Formatted at 2026-09-26 07:30:09 on dist-test-slave-wdjc"
server_key: "ff71b22018c3de9c16a43ee43cf4211d"
server_key_iv: "5ec4beb0d1a7d3517ee950c6087e4e65"
server_key_version: "encryptionkey@0"
I20260926 07:30:09.694911 27175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407783529717-22743-0/minicluster-data/ts-2/wal/instance:
uuid: "432b20d79d01448586e187be5b2e0c66"
format_stamp: "Formatted at 2026-09-26 07:30:09 on dist-test-slave-wdjc"
server_key: "ff71b22018c3de9c16a43ee43cf4211d"
server_key_iv: "5ec4beb0d1a7d3517ee950c6087e4e65"
server_key_version: "encryptionkey@0"
I20260926 07:30:09.699644 27175 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 07:30:09.702795 27189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:09.704638 27175 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:30:09.704802 27175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407783529717-22743-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407783529717-22743-0/minicluster-data/ts-2/wal
uuid: "432b20d79d01448586e187be5b2e0c66"
format_stamp: "Formatted at 2026-09-26 07:30:09 on dist-test-slave-wdjc"
server_key: "ff71b22018c3de9c16a43ee43cf4211d"
server_key_iv: "5ec4beb0d1a7d3517ee950c6087e4e65"
server_key_version: "encryptionkey@0"
I20260926 07:30:09.704949 27175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407783529717-22743-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407783529717-22743-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407783529717-22743-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:30:09.722931 27175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:30:09.727078 27175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:30:09.727355 27175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:30:09.728072 27175 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:30:09.729097 27175 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:30:09.729146 27175 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:09.729189 27175 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:30:09.729249 27175 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:09.739696 27175 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.195:34599
I20260926 07:30:09.739748 27302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.195:34599 every 8 connection(s)
I20260926 07:30:09.740448 27175 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407783529717-22743-0/minicluster-data/ts-2/data/info.pb
I20260926 07:30:09.744895 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 27175
I20260926 07:30:09.745085 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407783529717-22743-0/minicluster-data/ts-2/wal/instance
I20260926 07:30:09.745476 22743 external_mini_cluster.cc:1428] Setting key d55b980a32e9f4b63c8e14ce16de0b37
Sep 26 07:30:09 dist-test-slave-wdjc krb5kdc[26803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407809, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.195@KRBTEST.COM for kudu/127.22.53.254@KRBTEST.COM
I20260926 07:30:09.756709 26886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:09.742615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.195:49745 (local address 127.22.53.254:38209)
0926 07:30:09.742792 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:09.742799 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:09.744095 (+ 1296us) server_negotiation.cc:407] Connection header received
0926 07:30:09.745253 (+ 1158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:09.745257 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:09.745335 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:09.745458 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:09.746924 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:09.747557 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:09.749032 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:09.749273 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:09.749397 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:09.752327 (+ 2930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:09.752330 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:09.752332 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:09.752363 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:09.754179 (+ 1816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:09.754840 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:09.754845 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:09.754849 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:09.754949 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:09.755365 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:09.755369 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:09.755370 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:09.755598 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:09.755719 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:09.756315 (+ 596us) server_negotiation.cc:299] Negotiation successful
0926 07:30:09.756468 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 07:30:09.757630 27305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:09.743021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:38209 (local address 127.22.53.195:49745)
0926 07:30:09.743903 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:09.743959 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:09.744946 (+ 987us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:09.745678 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:09.745687 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:09.746135 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:09.746783 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:09.746799 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:09.747720 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:09.747723 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:09.748881 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:09.748896 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:09.749455 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:09.749459 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:09.749664 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:09.750324 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:09.750341 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:09.752085 (+ 1744us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:30:09.754348 (+ 2263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:09.754356 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:09.754377 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:09.754712 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:09.755055 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:09.755058 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:09.755060 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:09.755252 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:09.755719 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:30:09.755728 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:30:09.756028 (+ 300us) client_negotiation.cc:769] Sending connection context
0926 07:30:09.756271 (+ 243us) client_negotiation.cc:240] Negotiation successful
0926 07:30:09.756682 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":122,"threads_started":1}
I20260926 07:30:09.759119 27303 heartbeater.cc:347] Connected to a master server at 127.22.53.254:38209
I20260926 07:30:09.759411 27303 heartbeater.cc:464] Registering TS with master...
I20260926 07:30:09.760061 27303 heartbeater.cc:511] Master 127.22.53.254:38209 requested a full tablet report, sending...
I20260926 07:30:09.761296 26847 ts_manager.cc:194] Registered new tserver with Master: 432b20d79d01448586e187be5b2e0c66 (127.22.53.195:34599)
I20260926 07:30:09.761984 26847 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.53.195@KRBTEST.COM'} at 127.22.53.195:49745
I20260926 07:30:09.772948 22743 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:30:09.783845 26886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:09.775202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38390 (local address 127.22.53.254:38209)
0926 07:30:09.775375 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:09.775380 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:09.775578 (+ 198us) server_negotiation.cc:407] Connection header received
0926 07:30:09.775675 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:09.775678 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:09.775727 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:09.775802 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:09.776529 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:09.777139 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:09.778100 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:09.778336 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:09.778461 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:09.780269 (+ 1808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:09.780273 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:09.780275 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:09.780305 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:09.781920 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:09.782410 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:09.782413 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:09.782414 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:09.782497 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:09.782743 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:09.782746 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:09.782747 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:09.782989 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:09.783122 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:09.783465 (+ 343us) server_negotiation.cc:299] Negotiation successful
0926 07:30:09.783630 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 07:30:09.787830 26847 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38390:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:30:09.791265 26847 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:30:09.809244 27314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:09.803046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.194:33315 (local address 127.0.0.1:49438)
0926 07:30:09.803899 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:09.803913 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:09.803996 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:09.804773 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:09.804775 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:09.804796 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:30:09.804906 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:09.804912 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:09.806286 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:09.806293 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:09.807434 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:09.807444 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:09.808465 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:09.808468 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:09.808675 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:09.808775 (+ 100us) client_negotiation.cc:769] Sending connection context
0926 07:30:09.808882 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 07:30:09.809055 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":789,"mutex_wait_us":42,"thread_start_us":100,"threads_started":1}
I20260926 07:30:09.809302 27319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:09.803507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49438 (local address 127.22.53.194:33315)
0926 07:30:09.804254 (+ 747us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:09.804267 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:09.804289 (+ 22us) server_negotiation.cc:407] Connection header received
0926 07:30:09.804364 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:09.804371 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:09.804615 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:09.804773 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:30:09.805043 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:09.806151 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:09.807617 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:09.808281 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:09.808768 (+ 487us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:09.808937 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:09.808990 (+ 53us) server_negotiation.cc:299] Negotiation successful
0926 07:30:09.809108 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":647,"thread_start_us":119,"threads_started":1}
I20260926 07:30:09.809297 27315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:09.802275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53402 (local address 127.22.53.195:34599)
0926 07:30:09.802743 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:09.802758 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:09.803386 (+ 628us) server_negotiation.cc:407] Connection header received
0926 07:30:09.803743 (+ 357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:09.803748 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:09.803942 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:09.804076 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:30:09.804563 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:09.805858 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:09.807555 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:09.808248 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:09.808438 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:09.808736 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:09.808989 (+ 253us) server_negotiation.cc:299] Negotiation successful
0926 07:30:09.809108 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":186,"threads_started":1}
I20260926 07:30:09.810112 27316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:09.802010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.195:34599 (local address 127.0.0.1:53402)
0926 07:30:09.803218 (+ 1208us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:09.803280 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:09.803530 (+ 250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:09.804295 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:09.804297 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:09.804311 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:30:09.804411 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:09.804418 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:09.806013 (+ 1595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:09.806016 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:09.807379 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:09.807391 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:09.808445 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:09.808450 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:09.808710 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:09.808755 (+ 45us) client_negotiation.cc:769] Sending connection context
0926 07:30:09.809854 (+ 1099us) client_negotiation.cc:240] Negotiation successful
0926 07:30:09.809991 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1002,"mutex_wait_us":103,"thread_start_us":94,"threads_started":1}
I20260926 07:30:09.811854 27318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:09.802344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.193:39007 (local address 127.0.0.1:49268)
0926 07:30:09.803426 (+ 1082us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:09.803441 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:09.803564 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:09.804127 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:09.804135 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:09.804157 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:30:09.804304 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:09.804311 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:09.805750 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:09.805754 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:09.807378 (+ 1624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:09.807391 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:09.811380 (+ 3989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:09.811385 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:09.811500 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:09.811533 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 07:30:09.811607 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 07:30:09.811697 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1013,"mutex_wait_us":100,"thread_start_us":175,"threads_started":1}
I20260926 07:30:09.812867 27317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:09.802964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49268 (local address 127.22.53.193:39007)
0926 07:30:09.803392 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:09.803399 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:09.803541 (+ 142us) server_negotiation.cc:407] Connection header received
0926 07:30:09.803710 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:09.803715 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:09.803942 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:09.804467 (+ 525us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:30:09.804723 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:09.805621 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:09.808828 (+ 3207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:09.811202 (+ 2374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:09.812294 (+ 1092us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:09.812478 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:09.812579 (+ 101us) server_negotiation.cc:299] Negotiation successful
0926 07:30:09.812703 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":115,"threads_started":1}
I20260926 07:30:09.813182 27104 tablet_service.cc:1467] Processing CreateTablet for tablet 3b5c426106954134b3ce51a76f66f1bc (DEFAULT_TABLE table=test-table [id=d8da5df73338418898bd2a0357b2aefa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:30:09.813503 27237 tablet_service.cc:1467] Processing CreateTablet for tablet 3b5c426106954134b3ce51a76f66f1bc (DEFAULT_TABLE table=test-table [id=d8da5df73338418898bd2a0357b2aefa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:30:09.814600 27104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b5c426106954134b3ce51a76f66f1bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:30:09.814710 27237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b5c426106954134b3ce51a76f66f1bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:30:09.816334 26971 tablet_service.cc:1467] Processing CreateTablet for tablet 3b5c426106954134b3ce51a76f66f1bc (DEFAULT_TABLE table=test-table [id=d8da5df73338418898bd2a0357b2aefa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:30:09.817667 26971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b5c426106954134b3ce51a76f66f1bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:30:09.823205 27321 tablet_bootstrap.cc:492] T 3b5c426106954134b3ce51a76f66f1bc P 432b20d79d01448586e187be5b2e0c66: Bootstrap starting.
I20260926 07:30:09.825007 27320 tablet_bootstrap.cc:492] T 3b5c426106954134b3ce51a76f66f1bc P f62713828ed645318ecd9ab0e440ef72: Bootstrap starting.
I20260926 07:30:09.825703 27321 tablet_bootstrap.cc:654] T 3b5c426106954134b3ce51a76f66f1bc P 432b20d79d01448586e187be5b2e0c66: Neither blocks nor log segments found. Creating new log.
I20260926 07:30:09.826839 27322 tablet_bootstrap.cc:492] T 3b5c426106954134b3ce51a76f66f1bc P 49a79ddb38674ef9b143c90eaedd8c38: Bootstrap starting.
I20260926 07:30:09.826913 27321 log.cc:824] T 3b5c426106954134b3ce51a76f66f1bc P 432b20d79d01448586e187be5b2e0c66: Log is configured to *not* fsync() on all Append() calls
I20260926 07:30:09.827708 27320 tablet_bootstrap.cc:654] T 3b5c426106954134b3ce51a76f66f1bc P f62713828ed645318ecd9ab0e440ef72: Neither blocks nor log segments found. Creating new log.
I20260926 07:30:09.829154 27320 log.cc:824] T 3b5c426106954134b3ce51a76f66f1bc P f62713828ed645318ecd9ab0e440ef72: Log is configured to *not* fsync() on all Append() calls
I20260926 07:30:09.829500 27322 tablet_bootstrap.cc:654] T 3b5c426106954134b3ce51a76f66f1bc P 49a79ddb38674ef9b143c90eaedd8c38: Neither blocks nor log segments found. Creating new log.
I20260926 07:30:09.829960 27321 tablet_bootstrap.cc:492] T 3b5c426106954134b3ce51a76f66f1bc P 432b20d79d01448586e187be5b2e0c66: No bootstrap required, opened a new log
I20260926 07:30:09.830219 27321 ts_tablet_manager.cc:1397] T 3b5c426106954134b3ce51a76f66f1bc P 432b20d79d01448586e187be5b2e0c66: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260926 07:30:09.830459 27322 log.cc:824] T 3b5c426106954134b3ce51a76f66f1bc P 49a79ddb38674ef9b143c90eaedd8c38: Log is configured to *not* fsync() on all Append() calls
I20260926 07:30:09.831539 27320 tablet_bootstrap.cc:492] T 3b5c426106954134b3ce51a76f66f1bc P f62713828ed645318ecd9ab0e440ef72: No bootstrap required, opened a new log
I20260926 07:30:09.831770 27320 ts_tablet_manager.cc:1397] T 3b5c426106954134b3ce51a76f66f1bc P f62713828ed645318ecd9ab0e440ef72: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260926 07:30:09.832962 27322 tablet_bootstrap.cc:492] T 3b5c426106954134b3ce51a76f66f1bc P 49a79ddb38674ef9b143c90eaedd8c38: No bootstrap required, opened a new log
I20260926 07:30:09.833196 27322 ts_tablet_manager.cc:1397] T 3b5c426106954134b3ce51a76f66f1bc P 49a79ddb38674ef9b143c90eaedd8c38: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.001s
I20260926 07:30:09.834067 27321 raft_consensus.cc:348] T 3b5c426106954134b3ce51a76f66f1bc P 432b20d79d01448586e187be5b2e0c66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "432b20d79d01448586e187be5b2e0c66" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 34599 } } peers { permanent_uuid: "49a79ddb38674ef9b143c90eaedd8c38" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 39007 } } peers { permanent_uuid: "f62713828ed645318ecd9ab0e440ef72" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 33315 } }
I20260926 07:30:09.834390 27321 raft_consensus.cc:374] T 3b5c426106954134b3ce51a76f66f1bc P 432b20d79d01448586e187be5b2e0c66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:30:09.834450 27321 raft_consensus.cc:729] T 3b5c426106954134b3ce51a76f66f1bc P 432b20d79d01448586e187be5b2e0c66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 432b20d79d01448586e187be5b2e0c66, State: Initialized, Role: FOLLOWER
I20260926 07:30:09.835026 27321 consensus_queue.cc:260] T 3b5c426106954134b3ce51a76f66f1bc P 432b20d79d01448586e187be5b2e0c66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "432b20d79d01448586e187be5b2e0c66" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 34599 } } peers { permanent_uuid: "49a79ddb38674ef9b143c90eaedd8c38" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 39007 } } peers { permanent_uuid: "f62713828ed645318ecd9ab0e440ef72" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 33315 } }
I20260926 07:30:09.836330 27320 raft_consensus.cc:348] T 3b5c426106954134b3ce51a76f66f1bc P f62713828ed645318ecd9ab0e440ef72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "432b20d79d01448586e187be5b2e0c66" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 34599 } } peers { permanent_uuid: "49a79ddb38674ef9b143c90eaedd8c38" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 39007 } } peers { permanent_uuid: "f62713828ed645318ecd9ab0e440ef72" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 33315 } }
I20260926 07:30:09.836634 27321 ts_tablet_manager.cc:1428] T 3b5c426106954134b3ce51a76f66f1bc P 432b20d79d01448586e187be5b2e0c66: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260926 07:30:09.836707 27320 raft_consensus.cc:374] T 3b5c426106954134b3ce51a76f66f1bc P f62713828ed645318ecd9ab0e440ef72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:30:09.836777 27320 raft_consensus.cc:729] T 3b5c426106954134b3ce51a76f66f1bc P f62713828ed645318ecd9ab0e440ef72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f62713828ed645318ecd9ab0e440ef72, State: Initialized, Role: FOLLOWER
I20260926 07:30:09.837157 27303 heartbeater.cc:503] Master 127.22.53.254:38209 was elected leader, sending a full tablet report...
I20260926 07:30:09.837424 27320 consensus_queue.cc:260] T 3b5c426106954134b3ce51a76f66f1bc P f62713828ed645318ecd9ab0e440ef72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "432b20d79d01448586e187be5b2e0c66" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 34599 } } peers { permanent_uuid: "49a79ddb38674ef9b143c90eaedd8c38" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 39007 } } peers { permanent_uuid: "f62713828ed645318ecd9ab0e440ef72" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 33315 } }
I20260926 07:30:09.838258 27170 heartbeater.cc:503] Master 127.22.53.254:38209 was elected leader, sending a full tablet report...
I20260926 07:30:09.838305 27322 raft_consensus.cc:348] T 3b5c426106954134b3ce51a76f66f1bc P 49a79ddb38674ef9b143c90eaedd8c38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "432b20d79d01448586e187be5b2e0c66" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 34599 } } peers { permanent_uuid: "49a79ddb38674ef9b143c90eaedd8c38" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 39007 } } peers { permanent_uuid: "f62713828ed645318ecd9ab0e440ef72" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 33315 } }
I20260926 07:30:09.838609 27322 raft_consensus.cc:374] T 3b5c426106954134b3ce51a76f66f1bc P 49a79ddb38674ef9b143c90eaedd8c38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:30:09.838650 27320 ts_tablet_manager.cc:1428] T 3b5c426106954134b3ce51a76f66f1bc P f62713828ed645318ecd9ab0e440ef72: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260926 07:30:09.838678 27322 raft_consensus.cc:729] T 3b5c426106954134b3ce51a76f66f1bc P 49a79ddb38674ef9b143c90eaedd8c38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49a79ddb38674ef9b143c90eaedd8c38, State: Initialized, Role: FOLLOWER
I20260926 07:30:09.839164 27322 consensus_queue.cc:260] T 3b5c426106954134b3ce51a76f66f1bc P 49a79ddb38674ef9b143c90eaedd8c38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "432b20d79d01448586e187be5b2e0c66" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 34599 } } peers { permanent_uuid: "49a79ddb38674ef9b143c90eaedd8c38" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 39007 } } peers { permanent_uuid: "f62713828ed645318ecd9ab0e440ef72" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 33315 } }
I20260926 07:30:09.840059 27037 heartbeater.cc:503] Master 127.22.53.254:38209 was elected leader, sending a full tablet report...
I20260926 07:30:09.841185 27322 ts_tablet_manager.cc:1428] T 3b5c426106954134b3ce51a76f66f1bc P 49a79ddb38674ef9b143c90eaedd8c38: Time spent starting tablet: real 0.008s user 0.004s sys 0.003s
I20260926 07:30:09.910159 27326 raft_consensus.cc:482] T 3b5c426106954134b3ce51a76f66f1bc P 432b20d79d01448586e187be5b2e0c66 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:30:09.910430 27326 raft_consensus.cc:504] T 3b5c426106954134b3ce51a76f66f1bc P 432b20d79d01448586e187be5b2e0c66 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "432b20d79d01448586e187be5b2e0c66" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 34599 } } peers { permanent_uuid: "49a79ddb38674ef9b143c90eaedd8c38" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 39007 } } peers { permanent_uuid: "f62713828ed645318ecd9ab0e440ef72" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 33315 } }
I20260926 07:30:09.911660 27326 leader_election.cc:288] T 3b5c426106954134b3ce51a76f66f1bc P 432b20d79d01448586e187be5b2e0c66 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49a79ddb38674ef9b143c90eaedd8c38 (127.22.53.193:39007), f62713828ed645318ecd9ab0e440ef72 (127.22.53.194:33315)
I20260926 07:30:09.916282 27305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:09.912079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.193:39007 (local address 127.22.53.195:51531)
0926 07:30:09.912257 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:09.912279 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:09.912421 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:09.912839 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:09.912842 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:09.912866 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:30:09.913006 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:09.913013 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:09.913929 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:09.913933 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:09.914932 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:09.914943 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:09.915822 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:09.915826 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:09.915949 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:09.915976 (+ 27us) client_negotiation.cc:769] Sending connection context
0926 07:30:09.916054 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 07:30:09.916157 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59,"mutex_wait_us":50}
I20260926 07:30:09.916326 27317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:09.912174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.195:51531 (local address 127.22.53.193:39007)
0926 07:30:09.912348 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:09.912353 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:09.912373 (+ 20us) server_negotiation.cc:407] Connection header received
0926 07:30:09.912641 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:09.912646 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:09.912705 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:09.912799 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:30:09.913187 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:09.913805 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:09.915095 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:09.915680 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:09.915790 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:09.915836 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:09.916092 (+ 256us) server_negotiation.cc:299] Negotiation successful
0926 07:30:09.916182 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 07:30:09.916661 27319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:09.912627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.195:37009 (local address 127.22.53.194:33315)
0926 07:30:09.912803 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:09.912809 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:09.912905 (+ 96us) server_negotiation.cc:407] Connection header received
0926 07:30:09.913022 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:09.913025 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:09.913072 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:09.913154 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:30:09.913435 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:09.913989 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:09.915364 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:09.915854 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:09.915979 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:09.916043 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:09.916378 (+ 335us) server_negotiation.cc:299] Negotiation successful
0926 07:30:09.916473 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 07:30:09.916968 27329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:09.912392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.194:33315 (local address 127.22.53.195:37009)
0926 07:30:09.912819 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:09.912833 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:09.912927 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:09.913182 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:09.913185 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:09.913202 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:30:09.913322 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:09.913328 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:09.914150 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:09.914153 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:09.915072 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:09.915084 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:09.916101 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:09.916105 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:09.916228 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:09.916253 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 07:30:09.916338 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 07:30:09.916825 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"mutex_wait_us":2,"thread_start_us":168,"threads_started":1}
I20260926 07:30:09.917172 26991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b5c426106954134b3ce51a76f66f1bc" candidate_uuid: "432b20d79d01448586e187be5b2e0c66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49a79ddb38674ef9b143c90eaedd8c38" is_pre_election: true
I20260926 07:30:09.917554 26991 raft_consensus.cc:2454] T 3b5c426106954134b3ce51a76f66f1bc P 49a79ddb38674ef9b143c90eaedd8c38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 432b20d79d01448586e187be5b2e0c66 in term 0.
I20260926 07:30:09.917718 27124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b5c426106954134b3ce51a76f66f1bc" candidate_uuid: "432b20d79d01448586e187be5b2e0c66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f62713828ed645318ecd9ab0e440ef72" is_pre_election: true
I20260926 07:30:09.918028 27124 raft_consensus.cc:2454] T 3b5c426106954134b3ce51a76f66f1bc P f62713828ed645318ecd9ab0e440ef72 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 432b20d79d01448586e187be5b2e0c66 in term 0.
I20260926 07:30:09.918107 27191 leader_election.cc:302] T 3b5c426106954134b3ce51a76f66f1bc P 432b20d79d01448586e187be5b2e0c66 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 432b20d79d01448586e187be5b2e0c66, 49a79ddb38674ef9b143c90eaedd8c38; no voters:
I20260926 07:30:09.918466 27326 raft_consensus.cc:2781] T 3b5c426106954134b3ce51a76f66f1bc P 432b20d79d01448586e187be5b2e0c66 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:30:09.918708 27326 raft_consensus.cc:482] T 3b5c426106954134b3ce51a76f66f1bc P 432b20d79d01448586e187be5b2e0c66 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:30:09.918789 27326 raft_consensus.cc:3037] T 3b5c426106954134b3ce51a76f66f1bc P 432b20d79d01448586e187be5b2e0c66 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:30:09.919996 27326 raft_consensus.cc:504] T 3b5c426106954134b3ce51a76f66f1bc P 432b20d79d01448586e187be5b2e0c66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "432b20d79d01448586e187be5b2e0c66" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 34599 } } peers { permanent_uuid: "49a79ddb38674ef9b143c90eaedd8c38" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 39007 } } peers { permanent_uuid: "f62713828ed645318ecd9ab0e440ef72" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 33315 } }
I20260926 07:30:09.920403 27326 leader_election.cc:288] T 3b5c426106954134b3ce51a76f66f1bc P 432b20d79d01448586e187be5b2e0c66 [CANDIDATE]: Term 1 election: Requested vote from peers 49a79ddb38674ef9b143c90eaedd8c38 (127.22.53.193:39007), f62713828ed645318ecd9ab0e440ef72 (127.22.53.194:33315)
I20260926 07:30:09.921124 26991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b5c426106954134b3ce51a76f66f1bc" candidate_uuid: "432b20d79d01448586e187be5b2e0c66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49a79ddb38674ef9b143c90eaedd8c38"
I20260926 07:30:09.921237 27124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b5c426106954134b3ce51a76f66f1bc" candidate_uuid: "432b20d79d01448586e187be5b2e0c66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f62713828ed645318ecd9ab0e440ef72"
I20260926 07:30:09.921284 26991 raft_consensus.cc:3037] T 3b5c426106954134b3ce51a76f66f1bc P 49a79ddb38674ef9b143c90eaedd8c38 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:30:09.921420 27124 raft_consensus.cc:3037] T 3b5c426106954134b3ce51a76f66f1bc P f62713828ed645318ecd9ab0e440ef72 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:30:09.922511 26991 raft_consensus.cc:2454] T 3b5c426106954134b3ce51a76f66f1bc P 49a79ddb38674ef9b143c90eaedd8c38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 432b20d79d01448586e187be5b2e0c66 in term 1.
I20260926 07:30:09.922792 27124 raft_consensus.cc:2454] T 3b5c426106954134b3ce51a76f66f1bc P f62713828ed645318ecd9ab0e440ef72 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 432b20d79d01448586e187be5b2e0c66 in term 1.
I20260926 07:30:09.922914 27191 leader_election.cc:302] T 3b5c426106954134b3ce51a76f66f1bc P 432b20d79d01448586e187be5b2e0c66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 432b20d79d01448586e187be5b2e0c66, 49a79ddb38674ef9b143c90eaedd8c38; no voters:
I20260926 07:30:09.923270 27326 raft_consensus.cc:2781] T 3b5c426106954134b3ce51a76f66f1bc P 432b20d79d01448586e187be5b2e0c66 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:30:09.923517 27326 raft_consensus.cc:686] T 3b5c426106954134b3ce51a76f66f1bc P 432b20d79d01448586e187be5b2e0c66 [term 1 LEADER]: Becoming Leader. State: Replica: 432b20d79d01448586e187be5b2e0c66, State: Running, Role: LEADER
I20260926 07:30:09.923909 27326 consensus_queue.cc:237] T 3b5c426106954134b3ce51a76f66f1bc P 432b20d79d01448586e187be5b2e0c66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "432b20d79d01448586e187be5b2e0c66" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 34599 } } peers { permanent_uuid: "49a79ddb38674ef9b143c90eaedd8c38" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 39007 } } peers { permanent_uuid: "f62713828ed645318ecd9ab0e440ef72" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 33315 } }
I20260926 07:30:09.929174 26846 catalog_manager.cc:5495] T 3b5c426106954134b3ce51a76f66f1bc P 432b20d79d01448586e187be5b2e0c66 reported cstate change: term changed from 0 to 1, leader changed from <none> to 432b20d79d01448586e187be5b2e0c66 (127.22.53.195). New cstate: current_term: 1 leader_uuid: "432b20d79d01448586e187be5b2e0c66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "432b20d79d01448586e187be5b2e0c66" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 34599 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49a79ddb38674ef9b143c90eaedd8c38" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 39007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f62713828ed645318ecd9ab0e440ef72" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 33315 } health_report { overall_health: UNKNOWN } } }
W20260926 07:30:09.970383 27038 tablet.cc:2369] T 3b5c426106954134b3ce51a76f66f1bc P 49a79ddb38674ef9b143c90eaedd8c38: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:30:09.971777 27315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:09.967298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53414 (local address 127.22.53.195:34599)
0926 07:30:09.967440 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:09.967444 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:09.967459 (+ 15us) server_negotiation.cc:407] Connection header received
0926 07:30:09.967543 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:09.967545 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:09.967609 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:09.967713 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:30:09.968019 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:09.968596 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:09.969659 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:09.970152 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:09.970305 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:09.970709 (+ 404us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:30:09.971242 (+ 533us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:30:09.971414 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:09.971491 (+ 77us) server_negotiation.cc:299] Negotiation successful
0926 07:30:09.971569 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 07:30:09.979235 27124 raft_consensus.cc:1261] T 3b5c426106954134b3ce51a76f66f1bc P f62713828ed645318ecd9ab0e440ef72 [term 1 FOLLOWER]: Refusing update from remote peer 432b20d79d01448586e187be5b2e0c66: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:30:09.979235 26991 raft_consensus.cc:1261] T 3b5c426106954134b3ce51a76f66f1bc P 49a79ddb38674ef9b143c90eaedd8c38 [term 1 FOLLOWER]: Refusing update from remote peer 432b20d79d01448586e187be5b2e0c66: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:30:09.980161 27326 consensus_queue.cc:1035] T 3b5c426106954134b3ce51a76f66f1bc P 432b20d79d01448586e187be5b2e0c66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f62713828ed645318ecd9ab0e440ef72" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 33315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:30:09.980412 27330 consensus_queue.cc:1035] T 3b5c426106954134b3ce51a76f66f1bc P 432b20d79d01448586e187be5b2e0c66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49a79ddb38674ef9b143c90eaedd8c38" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 39007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 07:30:09.988252 27171 tablet.cc:2369] T 3b5c426106954134b3ce51a76f66f1bc P f62713828ed645318ecd9ab0e440ef72: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:30:09.991223 27334 mvcc.cc:205] Tried to move back new op lower bound from 7333510389666148352 to 7333510389453021184. Current Snapshot: MvccSnapshot[applied={T|T < 7333510389666148352}]
I20260926 07:30:10.000993 26846 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38390:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:30:10.001297 26846 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38390:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:30:10.004491 26846 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e65556604b494921bfba02647d624667: Sending DeleteTablet for 3 replicas of tablet 3b5c426106954134b3ce51a76f66f1bc
I20260926 07:30:10.005968 27237 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b5c426106954134b3ce51a76f66f1bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:30:10 UTC) from {username='kudu', principal='kudu/127.22.53.254@KRBTEST.COM'} at 127.0.0.1:53402
I20260926 07:30:10.006237 27104 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b5c426106954134b3ce51a76f66f1bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:30:10 UTC) from {username='kudu', principal='kudu/127.22.53.254@KRBTEST.COM'} at 127.0.0.1:49438
I20260926 07:30:10.006628 26971 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b5c426106954134b3ce51a76f66f1bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:30:10 UTC) from {username='kudu', principal='kudu/127.22.53.254@KRBTEST.COM'} at 127.0.0.1:49268
I20260926 07:30:10.007242 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 26909
I20260926 07:30:10.008041 27343 tablet_replica.cc:330] T 3b5c426106954134b3ce51a76f66f1bc P 432b20d79d01448586e187be5b2e0c66: stopping tablet replica
I20260926 07:30:10.008566 27343 raft_consensus.cc:2229] T 3b5c426106954134b3ce51a76f66f1bc P 432b20d79d01448586e187be5b2e0c66 [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:30:10.007108 27344 tablet_replica.cc:330] T 3b5c426106954134b3ce51a76f66f1bc P f62713828ed645318ecd9ab0e440ef72: stopping tablet replica
I20260926 07:30:10.009287 27344 raft_consensus.cc:2229] T 3b5c426106954134b3ce51a76f66f1bc P f62713828ed645318ecd9ab0e440ef72 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:30:10.009325 27343 raft_consensus.cc:2258] T 3b5c426106954134b3ce51a76f66f1bc P 432b20d79d01448586e187be5b2e0c66 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:30:10.009702 27344 raft_consensus.cc:2258] T 3b5c426106954134b3ce51a76f66f1bc P f62713828ed645318ecd9ab0e440ef72 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:30:10.011571 27344 ts_tablet_manager.cc:1905] T 3b5c426106954134b3ce51a76f66f1bc P f62713828ed645318ecd9ab0e440ef72: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:30:10.011833 27343 ts_tablet_manager.cc:1905] T 3b5c426106954134b3ce51a76f66f1bc P 432b20d79d01448586e187be5b2e0c66: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:30:10.015066 27344 ts_tablet_manager.cc:1918] T 3b5c426106954134b3ce51a76f66f1bc P f62713828ed645318ecd9ab0e440ef72: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:30:10.015218 27344 log.cc:1196] T 3b5c426106954134b3ce51a76f66f1bc P f62713828ed645318ecd9ab0e440ef72: Deleting WAL directory at /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407783529717-22743-0/minicluster-data/ts-1/wal/wals/3b5c426106954134b3ce51a76f66f1bc
I20260926 07:30:10.015584 27344 ts_tablet_manager.cc:1939] T 3b5c426106954134b3ce51a76f66f1bc P f62713828ed645318ecd9ab0e440ef72: Deleting consensus metadata
I20260926 07:30:10.015709 27343 ts_tablet_manager.cc:1918] T 3b5c426106954134b3ce51a76f66f1bc P 432b20d79d01448586e187be5b2e0c66: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:30:10.015822 27343 log.cc:1196] T 3b5c426106954134b3ce51a76f66f1bc P 432b20d79d01448586e187be5b2e0c66: Deleting WAL directory at /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407783529717-22743-0/minicluster-data/ts-2/wal/wals/3b5c426106954134b3ce51a76f66f1bc
I20260926 07:30:10.016149 27343 ts_tablet_manager.cc:1939] T 3b5c426106954134b3ce51a76f66f1bc P 432b20d79d01448586e187be5b2e0c66: Deleting consensus metadata
W20260926 07:30:10.017140 26832 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.53.193:39007 (error 108)
I20260926 07:30:10.017743 26831 catalog_manager.cc:4841] TS 432b20d79d01448586e187be5b2e0c66 (127.22.53.195:34599): tablet 3b5c426106954134b3ce51a76f66f1bc (table test-table [id=d8da5df73338418898bd2a0357b2aefa]) successfully deleted
I20260926 07:30:10.017743 26832 catalog_manager.cc:4841] TS f62713828ed645318ecd9ab0e440ef72 (127.22.53.194:33315): tablet 3b5c426106954134b3ce51a76f66f1bc (table test-table [id=d8da5df73338418898bd2a0357b2aefa]) successfully deleted
I20260926 07:30:10.018378 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 27042
I20260926 07:30:10.018638 27318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:10.018262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.193:39007 (local address 127.0.0.1:49282)
0926 07:30:10.018416 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:10.018537 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.193:39007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260926 07:30:10.020042 26832 catalog_manager.cc:4584] TS 49a79ddb38674ef9b143c90eaedd8c38 (127.22.53.193:39007): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3b5c426106954134b3ce51a76f66f1bc: Network error: Client connection negotiation failed: client connection to 127.22.53.193:39007: connect: Connection refused (error 111)
I20260926 07:30:10.027025 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 27175
I20260926 07:30:10.034168 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 26816
2026-09-26T07:30:10Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2577 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-26T07:30:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:30:10Z Disabled control of system clock
I20260926 07:30:10.068089 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407783529717-22743-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407783529717-22743-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407783529717-22743-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.53.254:42357
--webserver_interface=127.22.53.254
--webserver_port=0
--builtin_ntp_servers=127.22.53.212:39067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.53.254:42357
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 07:30:10.200960 27350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:30:10.201309 27350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:30:10.201351 27350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:30:10.206089 27350 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:30:10.206182 27350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:30:10.206198 27350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:30:10.206216 27350 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:30:10.206233 27350 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:30:10.211542 27350 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:39067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407783529717-22743-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407783529717-22743-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.53.254:42357
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.53.254:42357
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.53.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27350
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407783529717-22743-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:30:10.212814 27350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:30:10.214342 27350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:30:10.224238 27358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:10.224221 27356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:10.224239 27355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:10.225903 27350 server_base.cc:1034] running on GCE node
I20260926 07:30:10.226861 27350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:30:10.228760 27350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:30:10.230139 27350 hybrid_clock.cc:648] HybridClock initialized: now 1790407810230090 us; error 71 us; skew 500 ppm
I20260926 07:30:10.232846 27350 webserver.cc:460] Webserver started at http://127.22.53.254:43373/ using document root <none> and password file <none>
I20260926 07:30:10.233575 27350 fs_manager.cc:359] Metadata directory not provided
I20260926 07:30:10.233692 27350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:30:10.233958 27350 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:30:10.236617 27350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407783529717-22743-0/minicluster-data/master-0/data/instance:
uuid: "92567c7e97d0471581ac4cdc6d0b5194"
format_stamp: "Formatted at 2026-09-26 07:30:10 on dist-test-slave-wdjc"
server_key: "b58aa437c12f6f8323b5d77c532c42c6"
server_key_iv: "98da09d16463af260ab8a851c66902e3"
server_key_version: "encryptionkey@0"
I20260926 07:30:10.237368 27350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407783529717-22743-0/minicluster-data/master-0/wal/instance:
uuid: "92567c7e97d0471581ac4cdc6d0b5194"
format_stamp: "Formatted at 2026-09-26 07:30:10 on dist-test-slave-wdjc"
server_key: "b58aa437c12f6f8323b5d77c532c42c6"
server_key_iv: "98da09d16463af260ab8a851c66902e3"
server_key_version: "encryptionkey@0"
I20260926 07:30:10.242498 27350 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260926 07:30:10.245936 27364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:10.247526 27350 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:30:10.247735 27350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407783529717-22743-0/minicluster-data/master-0/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407783529717-22743-0/minicluster-data/master-0/wal
uuid: "92567c7e97d0471581ac4cdc6d0b5194"
format_stamp: "Formatted at 2026-09-26 07:30:10 on dist-test-slave-wdjc"
server_key: "b58aa437c12f6f8323b5d77c532c42c6"
server_key_iv: "98da09d16463af260ab8a851c66902e3"
server_key_version: "encryptionkey@0"
I20260926 07:30:10.247895 27350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407783529717-22743-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407783529717-22743-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407783529717-22743-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:30:10.254307 27350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:30:10.255100 27350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:30:10.255326 27350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:30:10.263206 27350 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.254:42357
I20260926 07:30:10.263284 27416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.254:42357 every 8 connection(s)
I20260926 07:30:10.264284 27350 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
I20260926 07:30:10.266288 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 27350
I20260926 07:30:10.266412 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407783529717-22743-0/minicluster-data/master-0/wal/instance
I20260926 07:30:10.266736 22743 external_mini_cluster.cc:1428] Setting key 9fa08e1deb0545a9099ffd56790668ec
I20260926 07:30:10.268890 27417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:30:10.275352 27420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:10.268303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56180 (local address 127.22.53.254:42357)
0926 07:30:10.269192 (+ 889us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:10.269219 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:10.269266 (+ 47us) server_negotiation.cc:407] Connection header received
0926 07:30:10.270355 (+ 1089us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:10.270382 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:10.271183 (+ 801us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:10.271785 (+ 602us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:10.273070 (+ 1285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:10.273077 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:10.273097 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:30:10.273113 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:10.273198 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:30:10.273228 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:10.273435 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:10.273785 (+ 350us) server_negotiation.cc:299] Negotiation successful
0926 07:30:10.274122 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":204,"threads_started":1}
I20260926 07:30:10.278749 27417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92567c7e97d0471581ac4cdc6d0b5194: Bootstrap starting.
I20260926 07:30:10.281630 27417 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 92567c7e97d0471581ac4cdc6d0b5194: Neither blocks nor log segments found. Creating new log.
I20260926 07:30:10.282785 27417 log.cc:824] T 00000000000000000000000000000000 P 92567c7e97d0471581ac4cdc6d0b5194: Log is configured to *not* fsync() on all Append() calls
I20260926 07:30:10.285208 27417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92567c7e97d0471581ac4cdc6d0b5194: No bootstrap required, opened a new log
I20260926 07:30:10.290035 27417 raft_consensus.cc:348] T 00000000000000000000000000000000 P 92567c7e97d0471581ac4cdc6d0b5194 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92567c7e97d0471581ac4cdc6d0b5194" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 42357 } }
I20260926 07:30:10.290378 27417 raft_consensus.cc:374] T 00000000000000000000000000000000 P 92567c7e97d0471581ac4cdc6d0b5194 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:30:10.290469 27417 raft_consensus.cc:729] T 00000000000000000000000000000000 P 92567c7e97d0471581ac4cdc6d0b5194 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92567c7e97d0471581ac4cdc6d0b5194, State: Initialized, Role: FOLLOWER
I20260926 07:30:10.291147 27417 consensus_queue.cc:260] T 00000000000000000000000000000000 P 92567c7e97d0471581ac4cdc6d0b5194 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92567c7e97d0471581ac4cdc6d0b5194" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 42357 } }
I20260926 07:30:10.291355 27417 raft_consensus.cc:388] T 00000000000000000000000000000000 P 92567c7e97d0471581ac4cdc6d0b5194 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:30:10.291450 27417 raft_consensus.cc:482] T 00000000000000000000000000000000 P 92567c7e97d0471581ac4cdc6d0b5194 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:30:10.291591 27417 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 92567c7e97d0471581ac4cdc6d0b5194 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:30:10.292919 27417 raft_consensus.cc:504] T 00000000000000000000000000000000 P 92567c7e97d0471581ac4cdc6d0b5194 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92567c7e97d0471581ac4cdc6d0b5194" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 42357 } }
I20260926 07:30:10.293352 27417 leader_election.cc:302] T 00000000000000000000000000000000 P 92567c7e97d0471581ac4cdc6d0b5194 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 92567c7e97d0471581ac4cdc6d0b5194; no voters:
I20260926 07:30:10.293790 27417 leader_election.cc:288] T 00000000000000000000000000000000 P 92567c7e97d0471581ac4cdc6d0b5194 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:30:10.294036 27422 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 92567c7e97d0471581ac4cdc6d0b5194 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:30:10.294385 27422 raft_consensus.cc:686] T 00000000000000000000000000000000 P 92567c7e97d0471581ac4cdc6d0b5194 [term 1 LEADER]: Becoming Leader. State: Replica: 92567c7e97d0471581ac4cdc6d0b5194, State: Running, Role: LEADER
I20260926 07:30:10.294838 27422 consensus_queue.cc:237] T 00000000000000000000000000000000 P 92567c7e97d0471581ac4cdc6d0b5194 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92567c7e97d0471581ac4cdc6d0b5194" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 42357 } }
I20260926 07:30:10.295312 27417 sys_catalog.cc:564] T 00000000000000000000000000000000 P 92567c7e97d0471581ac4cdc6d0b5194 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:30:10.296949 27423 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92567c7e97d0471581ac4cdc6d0b5194 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "92567c7e97d0471581ac4cdc6d0b5194" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92567c7e97d0471581ac4cdc6d0b5194" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 42357 } } }
I20260926 07:30:10.296952 27424 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92567c7e97d0471581ac4cdc6d0b5194 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 92567c7e97d0471581ac4cdc6d0b5194. Latest consensus state: current_term: 1 leader_uuid: "92567c7e97d0471581ac4cdc6d0b5194" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92567c7e97d0471581ac4cdc6d0b5194" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 42357 } } }
I20260926 07:30:10.297152 27424 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92567c7e97d0471581ac4cdc6d0b5194 [sys.catalog]: This master's current role is: LEADER
I20260926 07:30:10.297152 27423 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92567c7e97d0471581ac4cdc6d0b5194 [sys.catalog]: This master's current role is: LEADER
W20260926 07:30:10.301294 27437 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 92567c7e97d0471581ac4cdc6d0b5194: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:30:10.301414 27437 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:30:10.301622 27434 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:30:10.302461 27434 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:30:10.309461 27434 catalog_manager.cc:1347] Generated new cluster ID: e1b01359a26b4678abc6c03eac6a8b40
I20260926 07:30:10.309583 27434 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:30:10.320996 27434 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:30:10.321427 27434 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:30:10.329016 27434 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 92567c7e97d0471581ac4cdc6d0b5194: Generated new TSK 0
I20260926 07:30:10.329918 27434 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:30:10.341626 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407783529717-22743-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407783529717-22743-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407783529717-22743-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407783529717-22743-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.53.193:0
--local_ip_for_outbound_sockets=127.22.53.193
--webserver_interface=127.22.53.193
--webserver_port=0
--tserver_master_addrs=127.22.53.254:42357
--builtin_ntp_servers=127.22.53.212:39067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 07:30:10.488409 27441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:30:10.488802 27441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:30:10.488896 27441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:30:10.493690 27441 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:30:10.493788 27441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:30:10.493853 27441 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.53.193
I20260926 07:30:10.499943 27441 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:39067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407783529717-22743-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407783529717-22743-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.53.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407783529717-22743-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.53.193
--webserver_port=0
--tserver_master_addrs=127.22.53.254:42357
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27441
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.53.193
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407783529717-22743-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:30:10.501416 27441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:30:10.503044 27441 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:30:10.515074 27446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:10.515136 27447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:10.515074 27449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:10.515635 27441 server_base.cc:1034] running on GCE node
I20260926 07:30:10.516088 27441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:30:10.516939 27441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:30:10.518172 27441 hybrid_clock.cc:648] HybridClock initialized: now 1790407810518148 us; error 40 us; skew 500 ppm
I20260926 07:30:10.520627 27441 webserver.cc:460] Webserver started at http://127.22.53.193:41543/ using document root <none> and password file <none>
I20260926 07:30:10.521338 27441 fs_manager.cc:359] Metadata directory not provided
I20260926 07:30:10.521449 27441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:30:10.521703 27441 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:30:10.524130 27441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407783529717-22743-0/minicluster-data/ts-0/data/instance:
uuid: "bb6481aa262643f48d7ae2036ea2fab4"
format_stamp: "Formatted at 2026-09-26 07:30:10 on dist-test-slave-wdjc"
server_key: "34f432333d9e1f417eecd2c6b2347349"
server_key_iv: "140206f6c4a8d5427a1e0371137ee05d"
server_key_version: "encryptionkey@0"
I20260926 07:30:10.524863 27441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407783529717-22743-0/minicluster-data/ts-0/wal/instance:
uuid: "bb6481aa262643f48d7ae2036ea2fab4"
format_stamp: "Formatted at 2026-09-26 07:30:10 on dist-test-slave-wdjc"
server_key: "34f432333d9e1f417eecd2c6b2347349"
server_key_iv: "140206f6c4a8d5427a1e0371137ee05d"
server_key_version: "encryptionkey@0"
I20260926 07:30:10.529470 27441 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 07:30:10.532780 27455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:10.534579 27441 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 07:30:10.534773 27441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407783529717-22743-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407783529717-22743-0/minicluster-data/ts-0/wal
uuid: "bb6481aa262643f48d7ae2036ea2fab4"
format_stamp: "Formatted at 2026-09-26 07:30:10 on dist-test-slave-wdjc"
server_key: "34f432333d9e1f417eecd2c6b2347349"
server_key_iv: "140206f6c4a8d5427a1e0371137ee05d"
server_key_version: "encryptionkey@0"
I20260926 07:30:10.534936 27441 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407783529717-22743-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407783529717-22743-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407783529717-22743-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:30:10.540318 27441 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:30:10.541082 27441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:30:10.541299 27441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:30:10.541941 27441 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:30:10.542979 27441 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:30:10.543054 27441 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:10.543128 27441 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:30:10.543180 27441 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:10.553254 27441 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.193:41335
I20260926 07:30:10.553442 27568 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.193:41335 every 8 connection(s)
I20260926 07:30:10.554448 27441 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407783529717-22743-0/minicluster-data/ts-0/data/info.pb
I20260926 07:30:10.560949 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 27441
I20260926 07:30:10.561120 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407783529717-22743-0/minicluster-data/ts-0/wal/instance
I20260926 07:30:10.561517 22743 external_mini_cluster.cc:1428] Setting key 1ede181917b4356b54c6f8ec981e5963
I20260926 07:30:10.561658 27420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:10.556617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.193:51405 (local address 127.22.53.254:42357)
0926 07:30:10.556800 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:10.556805 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:10.558057 (+ 1252us) server_negotiation.cc:407] Connection header received
0926 07:30:10.559640 (+ 1583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:10.559644 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:10.559698 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:10.559786 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:10.561052 (+ 1266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:10.561056 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:10.561059 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:30:10.561073 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:10.561095 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:30:10.561108 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:10.561208 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:10.561447 (+ 239us) server_negotiation.cc:299] Negotiation successful
0926 07:30:10.561497 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 07:30:10.562939 27571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:10.557061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:42357 (local address 127.22.53.193:51405)
0926 07:30:10.557884 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:10.557933 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:10.559381 (+ 1448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:10.559980 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:10.559988 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:10.560090 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:10.560759 (+ 669us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:30:10.560761 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:10.560807 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:30:10.560886 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:30:10.561242 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:30:10.561254 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:30:10.561262 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 07:30:10.561492 (+ 230us) client_negotiation.cc:240] Negotiation successful
0926 07:30:10.561893 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":122,"threads_started":1}
I20260926 07:30:10.563839 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407783529717-22743-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407783529717-22743-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407783529717-22743-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407783529717-22743-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.53.194:0
--local_ip_for_outbound_sockets=127.22.53.194
--webserver_interface=127.22.53.194
--webserver_port=0
--tserver_master_addrs=127.22.53.254:42357
--builtin_ntp_servers=127.22.53.212:39067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260926 07:30:10.564322 27569 heartbeater.cc:347] Connected to a master server at 127.22.53.254:42357
I20260926 07:30:10.564646 27569 heartbeater.cc:464] Registering TS with master...
I20260926 07:30:10.565178 27569 heartbeater.cc:511] Master 127.22.53.254:42357 requested a full tablet report, sending...
I20260926 07:30:10.567042 27381 ts_manager.cc:194] Registered new tserver with Master: bb6481aa262643f48d7ae2036ea2fab4 (127.22.53.193:41335)
W20260926 07:30:10.687080 27572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:30:10.687470 27572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:30:10.687551 27572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:30:10.691887 27572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:30:10.692015 27572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:30:10.692123 27572 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.53.194
I20260926 07:30:10.698068 27572 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:39067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407783529717-22743-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407783529717-22743-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.53.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407783529717-22743-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.53.194
--webserver_port=0
--tserver_master_addrs=127.22.53.254:42357
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27572
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.53.194
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407783529717-22743-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:30:10.699483 27572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:30:10.701112 27572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:30:10.712826 27580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:10.712756 27578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:10.712756 27577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:10.712953 27572 server_base.cc:1034] running on GCE node
I20260926 07:30:10.713531 27572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:30:10.714371 27572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:30:10.715597 27572 hybrid_clock.cc:648] HybridClock initialized: now 1790407810715578 us; error 37 us; skew 500 ppm
I20260926 07:30:10.717980 27572 webserver.cc:460] Webserver started at http://127.22.53.194:45055/ using document root <none> and password file <none>
I20260926 07:30:10.718712 27572 fs_manager.cc:359] Metadata directory not provided
I20260926 07:30:10.718830 27572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:30:10.719110 27572 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:30:10.721670 27572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407783529717-22743-0/minicluster-data/ts-1/data/instance:
uuid: "3df449af48e6491382751ff909f2cafe"
format_stamp: "Formatted at 2026-09-26 07:30:10 on dist-test-slave-wdjc"
server_key: "12d823c21c546f3da7191824099f6bf5"
server_key_iv: "ebc3d7d90a3a78b46118fd3659049e73"
server_key_version: "encryptionkey@0"
I20260926 07:30:10.722394 27572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407783529717-22743-0/minicluster-data/ts-1/wal/instance:
uuid: "3df449af48e6491382751ff909f2cafe"
format_stamp: "Formatted at 2026-09-26 07:30:10 on dist-test-slave-wdjc"
server_key: "12d823c21c546f3da7191824099f6bf5"
server_key_iv: "ebc3d7d90a3a78b46118fd3659049e73"
server_key_version: "encryptionkey@0"
I20260926 07:30:10.727026 27572 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 07:30:10.730486 27586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:10.732672 27572 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 07:30:10.732882 27572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407783529717-22743-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407783529717-22743-0/minicluster-data/ts-1/wal
uuid: "3df449af48e6491382751ff909f2cafe"
format_stamp: "Formatted at 2026-09-26 07:30:10 on dist-test-slave-wdjc"
server_key: "12d823c21c546f3da7191824099f6bf5"
server_key_iv: "ebc3d7d90a3a78b46118fd3659049e73"
server_key_version: "encryptionkey@0"
I20260926 07:30:10.733058 27572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407783529717-22743-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407783529717-22743-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407783529717-22743-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:30:10.738456 27572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:30:10.739298 27572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:30:10.739562 27572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:30:10.740466 27572 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:30:10.741760 27572 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:30:10.741834 27572 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:10.741904 27572 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:30:10.741956 27572 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:10.752511 27572 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.194:45775
I20260926 07:30:10.752628 27699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.194:45775 every 8 connection(s)
I20260926 07:30:10.753750 27572 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407783529717-22743-0/minicluster-data/ts-1/data/info.pb
I20260926 07:30:10.762004 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 27572
I20260926 07:30:10.762099 27420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:10.755995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.194:44657 (local address 127.22.53.254:42357)
0926 07:30:10.756175 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:10.756180 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:10.757543 (+ 1363us) server_negotiation.cc:407] Connection header received
0926 07:30:10.759135 (+ 1592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:10.759138 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:10.759184 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:10.759273 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:10.760869 (+ 1596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:10.760874 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:10.760878 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:30:10.760925 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:10.760978 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:30:10.760994 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:10.761263 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:10.761841 (+ 578us) server_negotiation.cc:299] Negotiation successful
0926 07:30:10.761916 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 07:30:10.762166 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407783529717-22743-0/minicluster-data/ts-1/wal/instance
I20260926 07:30:10.762501 22743 external_mini_cluster.cc:1428] Setting key 38f209e8367e45178d33320e23b541df
I20260926 07:30:10.763240 27702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:10.756489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:42357 (local address 127.22.53.194:44657)
0926 07:30:10.757344 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:10.757392 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:10.758849 (+ 1457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:10.759480 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:10.759488 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:10.759595 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:10.760303 (+ 708us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:30:10.760305 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:10.760358 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:30:10.760443 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:30:10.761364 (+ 921us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:30:10.761386 (+ 22us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:30:10.761397 (+ 11us) client_negotiation.cc:769] Sending connection context
0926 07:30:10.761859 (+ 462us) client_negotiation.cc:240] Negotiation successful
0926 07:30:10.762186 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":127,"threads_started":1}
I20260926 07:30:10.764452 27700 heartbeater.cc:347] Connected to a master server at 127.22.53.254:42357
I20260926 07:30:10.764792 27700 heartbeater.cc:464] Registering TS with master...
I20260926 07:30:10.764814 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407783529717-22743-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407783529717-22743-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407783529717-22743-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407783529717-22743-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.53.195:0
--local_ip_for_outbound_sockets=127.22.53.195
--webserver_interface=127.22.53.195
--webserver_port=0
--tserver_master_addrs=127.22.53.254:42357
--builtin_ntp_servers=127.22.53.212:39067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260926 07:30:10.765303 27700 heartbeater.cc:511] Master 127.22.53.254:42357 requested a full tablet report, sending...
I20260926 07:30:10.766623 27381 ts_manager.cc:194] Registered new tserver with Master: 3df449af48e6491382751ff909f2cafe (127.22.53.194:45775)
W20260926 07:30:10.892974 27703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:30:10.893347 27703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:30:10.893400 27703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:30:10.898581 27703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:30:10.898726 27703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:30:10.898885 27703 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.53.195
I20260926 07:30:10.904680 27703 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:39067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407783529717-22743-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407783529717-22743-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.53.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407783529717-22743-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.53.195
--webserver_port=0
--tserver_master_addrs=127.22.53.254:42357
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27703
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.53.195
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407783529717-22743-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:30:10.905895 27703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:30:10.907395 27703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:30:10.919250 27709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:10.919270 27708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:10.919270 27711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:10.919751 27703 server_base.cc:1034] running on GCE node
I20260926 07:30:10.920403 27703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:30:10.921468 27703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:30:10.922748 27703 hybrid_clock.cc:648] HybridClock initialized: now 1790407810922714 us; error 55 us; skew 500 ppm
I20260926 07:30:10.925467 27703 webserver.cc:460] Webserver started at http://127.22.53.195:43353/ using document root <none> and password file <none>
I20260926 07:30:10.926263 27703 fs_manager.cc:359] Metadata directory not provided
I20260926 07:30:10.926383 27703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:30:10.926666 27703 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:30:10.929504 27703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407783529717-22743-0/minicluster-data/ts-2/data/instance:
uuid: "550f6c38624d48d8a73eeaca4d8c3a95"
format_stamp: "Formatted at 2026-09-26 07:30:10 on dist-test-slave-wdjc"
server_key: "c3fc24600c825126419eecd7e6ea820b"
server_key_iv: "22ff25f98f64165bdf56de382a500ab3"
server_key_version: "encryptionkey@0"
I20260926 07:30:10.930238 27703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407783529717-22743-0/minicluster-data/ts-2/wal/instance:
uuid: "550f6c38624d48d8a73eeaca4d8c3a95"
format_stamp: "Formatted at 2026-09-26 07:30:10 on dist-test-slave-wdjc"
server_key: "c3fc24600c825126419eecd7e6ea820b"
server_key_iv: "22ff25f98f64165bdf56de382a500ab3"
server_key_version: "encryptionkey@0"
I20260926 07:30:10.935274 27703 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 07:30:10.939793 27717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:10.941519 27703 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 07:30:10.941732 27703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407783529717-22743-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407783529717-22743-0/minicluster-data/ts-2/wal
uuid: "550f6c38624d48d8a73eeaca4d8c3a95"
format_stamp: "Formatted at 2026-09-26 07:30:10 on dist-test-slave-wdjc"
server_key: "c3fc24600c825126419eecd7e6ea820b"
server_key_iv: "22ff25f98f64165bdf56de382a500ab3"
server_key_version: "encryptionkey@0"
I20260926 07:30:10.941928 27703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407783529717-22743-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407783529717-22743-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407783529717-22743-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:30:10.947376 27703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:30:10.948166 27703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:30:10.948402 27703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:30:10.949110 27703 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:30:10.950212 27703 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:30:10.950291 27703 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:10.950368 27703 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:30:10.950420 27703 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:10.962275 27703 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.195:37919
I20260926 07:30:10.962302 27830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.195:37919 every 8 connection(s)
I20260926 07:30:10.963361 27703 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407783529717-22743-0/minicluster-data/ts-2/data/info.pb
I20260926 07:30:10.970652 27420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:10.965601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.195:51323 (local address 127.22.53.254:42357)
0926 07:30:10.965793 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:10.965798 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:10.967052 (+ 1254us) server_negotiation.cc:407] Connection header received
0926 07:30:10.968523 (+ 1471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:10.968527 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:10.968646 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:10.968736 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:10.970144 (+ 1408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:10.970147 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:10.970150 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:30:10.970163 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:10.970186 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:30:10.970194 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:10.970292 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:10.970475 (+ 183us) server_negotiation.cc:299] Negotiation successful
0926 07:30:10.970524 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260926 07:30:10.971647 27833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:10.966039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:42357 (local address 127.22.53.195:51323)
0926 07:30:10.966875 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:10.966924 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:10.968251 (+ 1327us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:10.968989 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:10.968997 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:10.969100 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:10.969767 (+ 667us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:30:10.969768 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:10.969815 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:30:10.969916 (+ 101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:30:10.970299 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:30:10.970307 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:30:10.970312 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 07:30:10.970484 (+ 172us) client_negotiation.cc:240] Negotiation successful
0926 07:30:10.970707 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":125,"threads_started":1}
I20260926 07:30:10.973163 27831 heartbeater.cc:347] Connected to a master server at 127.22.53.254:42357
I20260926 07:30:10.973420 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 27703
I20260926 07:30:10.973461 27831 heartbeater.cc:464] Registering TS with master...
I20260926 07:30:10.973570 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407783529717-22743-0/minicluster-data/ts-2/wal/instance
I20260926 07:30:10.973906 22743 external_mini_cluster.cc:1428] Setting key e9d60e4a26a87b0c6bb4c6fdccc0a821
I20260926 07:30:10.973999 27831 heartbeater.cc:511] Master 127.22.53.254:42357 requested a full tablet report, sending...
I20260926 07:30:10.975332 27381 ts_manager.cc:194] Registered new tserver with Master: 550f6c38624d48d8a73eeaca4d8c3a95 (127.22.53.195:37919)
I20260926 07:30:10.976408 22743 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:30:10.980657 27420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:10.978645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56188 (local address 127.22.53.254:42357)
0926 07:30:10.978846 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:10.978851 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:10.979024 (+ 173us) server_negotiation.cc:407] Connection header received
0926 07:30:10.979151 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:10.979155 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:10.979200 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:10.979295 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:10.980137 (+ 842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:10.980141 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:10.980144 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:30:10.980157 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:10.980179 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:30:10.980192 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:10.980307 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:10.980425 (+ 118us) server_negotiation.cc:299] Negotiation successful
0926 07:30:10.980473 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260926 07:30:10.984186 27380 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56188:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:30:10.987372 27380 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:30:11.001973 27845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:10.998753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.195:37919 (local address 127.0.0.1:57244)
0926 07:30:10.999676 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:10.999690 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:10.999780 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:11.000337 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:11.000345 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:11.000433 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:11.001094 (+ 661us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:30:11.001098 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:11.001123 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:30:11.001194 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:30:11.001588 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:30:11.001597 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:30:11.001604 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 07:30:11.001742 (+ 138us) client_negotiation.cc:240] Negotiation successful
0926 07:30:11.001813 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":856,"thread_start_us":88,"threads_started":1}
I20260926 07:30:11.002374 27846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:10.999150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.194:45775 (local address 127.0.0.1:40524)
0926 07:30:10.999966 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:10.999978 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:11.000088 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:11.001050 (+ 962us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:11.001053 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:11.001069 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:11.001523 (+ 454us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:30:11.001533 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:11.001548 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:30:11.001597 (+ 49us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:30:11.002124 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:30:11.002127 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:30:11.002130 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 07:30:11.002205 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 07:30:11.002240 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":759,"thread_start_us":93,"threads_started":1}
I20260926 07:30:11.002532 27842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:10.998321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.193:41335 (local address 127.0.0.1:37942)
0926 07:30:10.999590 (+ 1269us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:10.999643 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:10.999743 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:11.001402 (+ 1659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:11.001405 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:11.001418 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:11.001837 (+ 419us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:30:11.001838 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:11.001848 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:30:11.001877 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:30:11.002288 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:30:11.002294 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:30:11.002299 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 07:30:11.002367 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 07:30:11.002400 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1066,"thread_start_us":133,"threads_started":1}
I20260926 07:30:11.002663 27844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:10.998895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57244 (local address 127.22.53.195:37919)
0926 07:30:10.999416 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:10.999432 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:10.999842 (+ 410us) server_negotiation.cc:407] Connection header received
0926 07:30:10.999937 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:10.999943 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:11.000172 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:11.000314 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:11.001347 (+ 1033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:11.001356 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:11.001384 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:30:11.001396 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:11.001442 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:30:11.001476 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:11.002349 (+ 873us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:11.002414 (+ 65us) server_negotiation.cc:299] Negotiation successful
0926 07:30:11.002501 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":123,"threads_started":1}
I20260926 07:30:11.002944 27843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:10.998524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37942 (local address 127.22.53.193:41335)
0926 07:30:11.000337 (+ 1813us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:11.000349 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:11.000373 (+ 24us) server_negotiation.cc:407] Connection header received
0926 07:30:11.000421 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:11.000426 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:11.000618 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:11.000753 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:11.001998 (+ 1245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:11.002004 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:11.002014 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:30:11.002033 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:11.002098 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:30:11.002132 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:11.002243 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:11.002709 (+ 466us) server_negotiation.cc:299] Negotiation successful
0926 07:30:11.002790 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1690,"thread_start_us":115,"threads_started":1}
I20260926 07:30:11.002959 27847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:10.999307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40524 (local address 127.22.53.194:45775)
0926 07:30:10.999715 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:10.999728 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:11.000101 (+ 373us) server_negotiation.cc:407] Connection header received
0926 07:30:11.000179 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:11.000184 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:11.000345 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:11.000459 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:11.001779 (+ 1320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:11.001784 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:11.001795 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:30:11.001806 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:11.001845 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:30:11.001885 (+ 40us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:11.001987 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:11.002659 (+ 672us) server_negotiation.cc:299] Negotiation successful
0926 07:30:11.002802 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":128,"threads_started":1}
I20260926 07:30:11.005537 27765 tablet_service.cc:1467] Processing CreateTablet for tablet 63fc05ed36b34b11b79efdfb43f2012a (DEFAULT_TABLE table=test-table [id=343dd9c8f61e49319dd28f7f630f9e2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:30:11.005607 27503 tablet_service.cc:1467] Processing CreateTablet for tablet 63fc05ed36b34b11b79efdfb43f2012a (DEFAULT_TABLE table=test-table [id=343dd9c8f61e49319dd28f7f630f9e2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:30:11.006672 27503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63fc05ed36b34b11b79efdfb43f2012a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:30:11.006672 27765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63fc05ed36b34b11b79efdfb43f2012a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:30:11.006470 27634 tablet_service.cc:1467] Processing CreateTablet for tablet 63fc05ed36b34b11b79efdfb43f2012a (DEFAULT_TABLE table=test-table [id=343dd9c8f61e49319dd28f7f630f9e2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:30:11.007481 27634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63fc05ed36b34b11b79efdfb43f2012a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:30:11.013649 27848 tablet_bootstrap.cc:492] T 63fc05ed36b34b11b79efdfb43f2012a P 550f6c38624d48d8a73eeaca4d8c3a95: Bootstrap starting.
I20260926 07:30:11.016328 27850 tablet_bootstrap.cc:492] T 63fc05ed36b34b11b79efdfb43f2012a P 3df449af48e6491382751ff909f2cafe: Bootstrap starting.
I20260926 07:30:11.016829 27848 tablet_bootstrap.cc:654] T 63fc05ed36b34b11b79efdfb43f2012a P 550f6c38624d48d8a73eeaca4d8c3a95: Neither blocks nor log segments found. Creating new log.
I20260926 07:30:11.017138 27849 tablet_bootstrap.cc:492] T 63fc05ed36b34b11b79efdfb43f2012a P bb6481aa262643f48d7ae2036ea2fab4: Bootstrap starting.
I20260926 07:30:11.017937 27848 log.cc:824] T 63fc05ed36b34b11b79efdfb43f2012a P 550f6c38624d48d8a73eeaca4d8c3a95: Log is configured to *not* fsync() on all Append() calls
I20260926 07:30:11.018854 27850 tablet_bootstrap.cc:654] T 63fc05ed36b34b11b79efdfb43f2012a P 3df449af48e6491382751ff909f2cafe: Neither blocks nor log segments found. Creating new log.
I20260926 07:30:11.019443 27849 tablet_bootstrap.cc:654] T 63fc05ed36b34b11b79efdfb43f2012a P bb6481aa262643f48d7ae2036ea2fab4: Neither blocks nor log segments found. Creating new log.
I20260926 07:30:11.019723 27850 log.cc:824] T 63fc05ed36b34b11b79efdfb43f2012a P 3df449af48e6491382751ff909f2cafe: Log is configured to *not* fsync() on all Append() calls
I20260926 07:30:11.020247 27849 log.cc:824] T 63fc05ed36b34b11b79efdfb43f2012a P bb6481aa262643f48d7ae2036ea2fab4: Log is configured to *not* fsync() on all Append() calls
I20260926 07:30:11.020305 27848 tablet_bootstrap.cc:492] T 63fc05ed36b34b11b79efdfb43f2012a P 550f6c38624d48d8a73eeaca4d8c3a95: No bootstrap required, opened a new log
I20260926 07:30:11.020581 27848 ts_tablet_manager.cc:1397] T 63fc05ed36b34b11b79efdfb43f2012a P 550f6c38624d48d8a73eeaca4d8c3a95: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260926 07:30:11.022403 27849 tablet_bootstrap.cc:492] T 63fc05ed36b34b11b79efdfb43f2012a P bb6481aa262643f48d7ae2036ea2fab4: No bootstrap required, opened a new log
I20260926 07:30:11.022562 27850 tablet_bootstrap.cc:492] T 63fc05ed36b34b11b79efdfb43f2012a P 3df449af48e6491382751ff909f2cafe: No bootstrap required, opened a new log
I20260926 07:30:11.022644 27849 ts_tablet_manager.cc:1397] T 63fc05ed36b34b11b79efdfb43f2012a P bb6481aa262643f48d7ae2036ea2fab4: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 07:30:11.022802 27850 ts_tablet_manager.cc:1397] T 63fc05ed36b34b11b79efdfb43f2012a P 3df449af48e6491382751ff909f2cafe: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260926 07:30:11.025827 27848 raft_consensus.cc:348] T 63fc05ed36b34b11b79efdfb43f2012a P 550f6c38624d48d8a73eeaca4d8c3a95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb6481aa262643f48d7ae2036ea2fab4" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 41335 } } peers { permanent_uuid: "3df449af48e6491382751ff909f2cafe" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 45775 } } peers { permanent_uuid: "550f6c38624d48d8a73eeaca4d8c3a95" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 37919 } }
I20260926 07:30:11.026240 27848 raft_consensus.cc:374] T 63fc05ed36b34b11b79efdfb43f2012a P 550f6c38624d48d8a73eeaca4d8c3a95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:30:11.026300 27848 raft_consensus.cc:729] T 63fc05ed36b34b11b79efdfb43f2012a P 550f6c38624d48d8a73eeaca4d8c3a95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 550f6c38624d48d8a73eeaca4d8c3a95, State: Initialized, Role: FOLLOWER
I20260926 07:30:11.026105 27849 raft_consensus.cc:348] T 63fc05ed36b34b11b79efdfb43f2012a P bb6481aa262643f48d7ae2036ea2fab4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb6481aa262643f48d7ae2036ea2fab4" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 41335 } } peers { permanent_uuid: "3df449af48e6491382751ff909f2cafe" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 45775 } } peers { permanent_uuid: "550f6c38624d48d8a73eeaca4d8c3a95" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 37919 } }
I20260926 07:30:11.026363 27849 raft_consensus.cc:374] T 63fc05ed36b34b11b79efdfb43f2012a P bb6481aa262643f48d7ae2036ea2fab4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:30:11.026422 27849 raft_consensus.cc:729] T 63fc05ed36b34b11b79efdfb43f2012a P bb6481aa262643f48d7ae2036ea2fab4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb6481aa262643f48d7ae2036ea2fab4, State: Initialized, Role: FOLLOWER
I20260926 07:30:11.026820 27849 consensus_queue.cc:260] T 63fc05ed36b34b11b79efdfb43f2012a P bb6481aa262643f48d7ae2036ea2fab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb6481aa262643f48d7ae2036ea2fab4" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 41335 } } peers { permanent_uuid: "3df449af48e6491382751ff909f2cafe" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 45775 } } peers { permanent_uuid: "550f6c38624d48d8a73eeaca4d8c3a95" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 37919 } }
I20260926 07:30:11.026804 27848 consensus_queue.cc:260] T 63fc05ed36b34b11b79efdfb43f2012a P 550f6c38624d48d8a73eeaca4d8c3a95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb6481aa262643f48d7ae2036ea2fab4" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 41335 } } peers { permanent_uuid: "3df449af48e6491382751ff909f2cafe" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 45775 } } peers { permanent_uuid: "550f6c38624d48d8a73eeaca4d8c3a95" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 37919 } }
I20260926 07:30:11.027593 27831 heartbeater.cc:503] Master 127.22.53.254:42357 was elected leader, sending a full tablet report...
I20260926 07:30:11.027856 27569 heartbeater.cc:503] Master 127.22.53.254:42357 was elected leader, sending a full tablet report...
I20260926 07:30:11.027951 27849 ts_tablet_manager.cc:1428] T 63fc05ed36b34b11b79efdfb43f2012a P bb6481aa262643f48d7ae2036ea2fab4: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260926 07:30:11.028820 27848 ts_tablet_manager.cc:1428] T 63fc05ed36b34b11b79efdfb43f2012a P 550f6c38624d48d8a73eeaca4d8c3a95: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260926 07:30:11.028661 27850 raft_consensus.cc:348] T 63fc05ed36b34b11b79efdfb43f2012a P 3df449af48e6491382751ff909f2cafe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb6481aa262643f48d7ae2036ea2fab4" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 41335 } } peers { permanent_uuid: "3df449af48e6491382751ff909f2cafe" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 45775 } } peers { permanent_uuid: "550f6c38624d48d8a73eeaca4d8c3a95" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 37919 } }
I20260926 07:30:11.028931 27850 raft_consensus.cc:374] T 63fc05ed36b34b11b79efdfb43f2012a P 3df449af48e6491382751ff909f2cafe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:30:11.028978 27850 raft_consensus.cc:729] T 63fc05ed36b34b11b79efdfb43f2012a P 3df449af48e6491382751ff909f2cafe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3df449af48e6491382751ff909f2cafe, State: Initialized, Role: FOLLOWER
I20260926 07:30:11.029438 27850 consensus_queue.cc:260] T 63fc05ed36b34b11b79efdfb43f2012a P 3df449af48e6491382751ff909f2cafe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb6481aa262643f48d7ae2036ea2fab4" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 41335 } } peers { permanent_uuid: "3df449af48e6491382751ff909f2cafe" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 45775 } } peers { permanent_uuid: "550f6c38624d48d8a73eeaca4d8c3a95" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 37919 } }
I20260926 07:30:11.030174 27700 heartbeater.cc:503] Master 127.22.53.254:42357 was elected leader, sending a full tablet report...
I20260926 07:30:11.030571 27850 ts_tablet_manager.cc:1428] T 63fc05ed36b34b11b79efdfb43f2012a P 3df449af48e6491382751ff909f2cafe: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
W20260926 07:30:11.055821 27570 tablet.cc:2369] T 63fc05ed36b34b11b79efdfb43f2012a P bb6481aa262643f48d7ae2036ea2fab4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:30:11.184119 27856 raft_consensus.cc:482] T 63fc05ed36b34b11b79efdfb43f2012a P 3df449af48e6491382751ff909f2cafe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:30:11.184374 27856 raft_consensus.cc:504] T 63fc05ed36b34b11b79efdfb43f2012a P 3df449af48e6491382751ff909f2cafe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb6481aa262643f48d7ae2036ea2fab4" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 41335 } } peers { permanent_uuid: "3df449af48e6491382751ff909f2cafe" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 45775 } } peers { permanent_uuid: "550f6c38624d48d8a73eeaca4d8c3a95" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 37919 } }
I20260926 07:30:11.185727 27856 leader_election.cc:288] T 63fc05ed36b34b11b79efdfb43f2012a P 3df449af48e6491382751ff909f2cafe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb6481aa262643f48d7ae2036ea2fab4 (127.22.53.193:41335), 550f6c38624d48d8a73eeaca4d8c3a95 (127.22.53.195:37919)
I20260926 07:30:11.188015 27843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:11.186255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.194:50557 (local address 127.22.53.193:41335)
0926 07:30:11.186385 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:11.186392 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:11.186555 (+ 163us) server_negotiation.cc:407] Connection header received
0926 07:30:11.186631 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:11.186634 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:11.186672 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:11.186750 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:11.187546 (+ 796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:11.187550 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:11.187552 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:30:11.187567 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:11.187591 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:30:11.187602 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:11.187706 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:11.187817 (+ 111us) server_negotiation.cc:299] Negotiation successful
0926 07:30:11.187861 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 07:30:11.188001 27844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:11.186148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.194:33761 (local address 127.22.53.195:37919)
0926 07:30:11.186279 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:11.186284 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:11.186305 (+ 21us) server_negotiation.cc:407] Connection header received
0926 07:30:11.186464 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:11.186467 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:11.186502 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:11.186612 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:11.187552 (+ 940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:11.187556 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:11.187558 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:30:11.187570 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:11.187591 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:30:11.187602 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:11.187706 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:11.187824 (+ 118us) server_negotiation.cc:299] Negotiation successful
0926 07:30:11.187861 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 07:30:11.188002 27702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:11.186028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.195:37919 (local address 127.22.53.194:33761)
0926 07:30:11.186194 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:11.186214 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:11.186320 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:11.186663 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:11.186665 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:11.186677 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:11.187294 (+ 617us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:30:11.187299 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:11.187316 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:30:11.187363 (+ 47us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:30:11.187710 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:30:11.187714 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:30:11.187718 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 07:30:11.187810 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 07:30:11.187851 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260926 07:30:11.188014 27857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:11.186065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.193:41335 (local address 127.22.53.194:50557)
0926 07:30:11.186471 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:11.186482 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:11.186545 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:11.186793 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:11.186798 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:11.186814 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:11.187349 (+ 535us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:30:11.187350 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:11.187358 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:30:11.187373 (+ 15us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:30:11.187710 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:30:11.187714 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:30:11.187717 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 07:30:11.187810 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 07:30:11.187878 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":117,"threads_started":1}
I20260926 07:30:11.188699 27523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63fc05ed36b34b11b79efdfb43f2012a" candidate_uuid: "3df449af48e6491382751ff909f2cafe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb6481aa262643f48d7ae2036ea2fab4" is_pre_election: true
I20260926 07:30:11.188812 27785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63fc05ed36b34b11b79efdfb43f2012a" candidate_uuid: "3df449af48e6491382751ff909f2cafe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "550f6c38624d48d8a73eeaca4d8c3a95" is_pre_election: true
I20260926 07:30:11.189069 27523 raft_consensus.cc:2454] T 63fc05ed36b34b11b79efdfb43f2012a P bb6481aa262643f48d7ae2036ea2fab4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3df449af48e6491382751ff909f2cafe in term 0.
I20260926 07:30:11.189107 27785 raft_consensus.cc:2454] T 63fc05ed36b34b11b79efdfb43f2012a P 550f6c38624d48d8a73eeaca4d8c3a95 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3df449af48e6491382751ff909f2cafe in term 0.
I20260926 07:30:11.189628 27590 leader_election.cc:302] T 63fc05ed36b34b11b79efdfb43f2012a P 3df449af48e6491382751ff909f2cafe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3df449af48e6491382751ff909f2cafe, 550f6c38624d48d8a73eeaca4d8c3a95, bb6481aa262643f48d7ae2036ea2fab4; no voters:
I20260926 07:30:11.189963 27856 raft_consensus.cc:2781] T 63fc05ed36b34b11b79efdfb43f2012a P 3df449af48e6491382751ff909f2cafe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:30:11.190070 27856 raft_consensus.cc:482] T 63fc05ed36b34b11b79efdfb43f2012a P 3df449af48e6491382751ff909f2cafe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:30:11.190132 27856 raft_consensus.cc:3037] T 63fc05ed36b34b11b79efdfb43f2012a P 3df449af48e6491382751ff909f2cafe [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:30:11.191268 27856 raft_consensus.cc:504] T 63fc05ed36b34b11b79efdfb43f2012a P 3df449af48e6491382751ff909f2cafe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb6481aa262643f48d7ae2036ea2fab4" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 41335 } } peers { permanent_uuid: "3df449af48e6491382751ff909f2cafe" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 45775 } } peers { permanent_uuid: "550f6c38624d48d8a73eeaca4d8c3a95" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 37919 } }
I20260926 07:30:11.191700 27856 leader_election.cc:288] T 63fc05ed36b34b11b79efdfb43f2012a P 3df449af48e6491382751ff909f2cafe [CANDIDATE]: Term 1 election: Requested vote from peers bb6481aa262643f48d7ae2036ea2fab4 (127.22.53.193:41335), 550f6c38624d48d8a73eeaca4d8c3a95 (127.22.53.195:37919)
I20260926 07:30:11.192162 27785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63fc05ed36b34b11b79efdfb43f2012a" candidate_uuid: "3df449af48e6491382751ff909f2cafe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "550f6c38624d48d8a73eeaca4d8c3a95"
I20260926 07:30:11.192305 27785 raft_consensus.cc:3037] T 63fc05ed36b34b11b79efdfb43f2012a P 550f6c38624d48d8a73eeaca4d8c3a95 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:30:11.192764 27523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63fc05ed36b34b11b79efdfb43f2012a" candidate_uuid: "3df449af48e6491382751ff909f2cafe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb6481aa262643f48d7ae2036ea2fab4"
I20260926 07:30:11.192941 27523 raft_consensus.cc:3037] T 63fc05ed36b34b11b79efdfb43f2012a P bb6481aa262643f48d7ae2036ea2fab4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:30:11.193668 27785 raft_consensus.cc:2454] T 63fc05ed36b34b11b79efdfb43f2012a P 550f6c38624d48d8a73eeaca4d8c3a95 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3df449af48e6491382751ff909f2cafe in term 1.
I20260926 07:30:11.194051 27590 leader_election.cc:302] T 63fc05ed36b34b11b79efdfb43f2012a P 3df449af48e6491382751ff909f2cafe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3df449af48e6491382751ff909f2cafe, 550f6c38624d48d8a73eeaca4d8c3a95; no voters:
I20260926 07:30:11.194283 27856 raft_consensus.cc:2781] T 63fc05ed36b34b11b79efdfb43f2012a P 3df449af48e6491382751ff909f2cafe [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:30:11.194312 27523 raft_consensus.cc:2454] T 63fc05ed36b34b11b79efdfb43f2012a P bb6481aa262643f48d7ae2036ea2fab4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3df449af48e6491382751ff909f2cafe in term 1.
I20260926 07:30:11.194559 27856 raft_consensus.cc:686] T 63fc05ed36b34b11b79efdfb43f2012a P 3df449af48e6491382751ff909f2cafe [term 1 LEADER]: Becoming Leader. State: Replica: 3df449af48e6491382751ff909f2cafe, State: Running, Role: LEADER
I20260926 07:30:11.195008 27856 consensus_queue.cc:237] T 63fc05ed36b34b11b79efdfb43f2012a P 3df449af48e6491382751ff909f2cafe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb6481aa262643f48d7ae2036ea2fab4" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 41335 } } peers { permanent_uuid: "3df449af48e6491382751ff909f2cafe" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 45775 } } peers { permanent_uuid: "550f6c38624d48d8a73eeaca4d8c3a95" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 37919 } }
I20260926 07:30:11.198951 27381 catalog_manager.cc:5495] T 63fc05ed36b34b11b79efdfb43f2012a P 3df449af48e6491382751ff909f2cafe reported cstate change: term changed from 0 to 1, leader changed from <none> to 3df449af48e6491382751ff909f2cafe (127.22.53.194). New cstate: current_term: 1 leader_uuid: "3df449af48e6491382751ff909f2cafe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb6481aa262643f48d7ae2036ea2fab4" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 41335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3df449af48e6491382751ff909f2cafe" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 45775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "550f6c38624d48d8a73eeaca4d8c3a95" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 37919 } health_report { overall_health: UNKNOWN } } }
W20260926 07:30:11.214567 27832 tablet.cc:2369] T 63fc05ed36b34b11b79efdfb43f2012a P 550f6c38624d48d8a73eeaca4d8c3a95: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:30:11.245265 27847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:11.243455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40540 (local address 127.22.53.194:45775)
0926 07:30:11.243620 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:11.243628 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:11.243644 (+ 16us) server_negotiation.cc:407] Connection header received
0926 07:30:11.243745 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:11.243747 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:11.243819 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:11.243927 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:11.244782 (+ 855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:11.244785 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:11.244787 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:30:11.244798 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:11.244820 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:30:11.244829 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:11.244937 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:11.245062 (+ 125us) server_negotiation.cc:299] Negotiation successful
0926 07:30:11.245106 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 07:30:11.252838 27785 raft_consensus.cc:1261] T 63fc05ed36b34b11b79efdfb43f2012a P 550f6c38624d48d8a73eeaca4d8c3a95 [term 1 FOLLOWER]: Refusing update from remote peer 3df449af48e6491382751ff909f2cafe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:30:11.252837 27523 raft_consensus.cc:1261] T 63fc05ed36b34b11b79efdfb43f2012a P bb6481aa262643f48d7ae2036ea2fab4 [term 1 FOLLOWER]: Refusing update from remote peer 3df449af48e6491382751ff909f2cafe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:30:11.253822 27858 consensus_queue.cc:1035] T 63fc05ed36b34b11b79efdfb43f2012a P 3df449af48e6491382751ff909f2cafe [LEADER]: Connected to new peer: Peer: permanent_uuid: "550f6c38624d48d8a73eeaca4d8c3a95" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 37919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:30:11.254071 27856 consensus_queue.cc:1035] T 63fc05ed36b34b11b79efdfb43f2012a P 3df449af48e6491382751ff909f2cafe [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb6481aa262643f48d7ae2036ea2fab4" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 41335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 07:30:11.254993 27701 tablet.cc:2369] T 63fc05ed36b34b11b79efdfb43f2012a P 3df449af48e6491382751ff909f2cafe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:30:11.262549 27861 mvcc.cc:205] Tried to move back new op lower bound from 7333510394882457600 to 7333510394657820672. Current Snapshot: MvccSnapshot[applied={T|T < 7333510394882457600}]
I20260926 07:30:11.262850 27863 mvcc.cc:205] Tried to move back new op lower bound from 7333510394882457600 to 7333510394657820672. Current Snapshot: MvccSnapshot[applied={T|T < 7333510394882457600}]
I20260926 07:30:11.275610 27380 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56188:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:30:11.275928 27380 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56188:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:30:11.279285 27380 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 92567c7e97d0471581ac4cdc6d0b5194: Sending DeleteTablet for 3 replicas of tablet 63fc05ed36b34b11b79efdfb43f2012a
I20260926 07:30:11.280416 27765 tablet_service.cc:1514] Processing DeleteTablet for tablet 63fc05ed36b34b11b79efdfb43f2012a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:30:11 UTC) from {username='slave'} at 127.0.0.1:57244
I20260926 07:30:11.280453 27503 tablet_service.cc:1514] Processing DeleteTablet for tablet 63fc05ed36b34b11b79efdfb43f2012a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:30:11 UTC) from {username='slave'} at 127.0.0.1:37942
I20260926 07:30:11.280628 27634 tablet_service.cc:1514] Processing DeleteTablet for tablet 63fc05ed36b34b11b79efdfb43f2012a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:30:11 UTC) from {username='slave'} at 127.0.0.1:40524
I20260926 07:30:11.281121 27871 tablet_replica.cc:330] T 63fc05ed36b34b11b79efdfb43f2012a P 550f6c38624d48d8a73eeaca4d8c3a95: stopping tablet replica
I20260926 07:30:11.281412 27872 tablet_replica.cc:330] T 63fc05ed36b34b11b79efdfb43f2012a P bb6481aa262643f48d7ae2036ea2fab4: stopping tablet replica
I20260926 07:30:11.281881 27872 raft_consensus.cc:2229] T 63fc05ed36b34b11b79efdfb43f2012a P bb6481aa262643f48d7ae2036ea2fab4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:30:11.282171 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 27441
I20260926 07:30:11.282388 27871 raft_consensus.cc:2229] T 63fc05ed36b34b11b79efdfb43f2012a P 550f6c38624d48d8a73eeaca4d8c3a95 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:30:11.282850 27871 raft_consensus.cc:2258] T 63fc05ed36b34b11b79efdfb43f2012a P 550f6c38624d48d8a73eeaca4d8c3a95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:30:11.282977 27873 tablet_replica.cc:330] T 63fc05ed36b34b11b79efdfb43f2012a P 3df449af48e6491382751ff909f2cafe: stopping tablet replica
I20260926 07:30:11.283296 27873 raft_consensus.cc:2229] T 63fc05ed36b34b11b79efdfb43f2012a P 3df449af48e6491382751ff909f2cafe [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:30:11.283775 27873 raft_consensus.cc:2258] T 63fc05ed36b34b11b79efdfb43f2012a P 3df449af48e6491382751ff909f2cafe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:30:11.284994 27871 ts_tablet_manager.cc:1905] T 63fc05ed36b34b11b79efdfb43f2012a P 550f6c38624d48d8a73eeaca4d8c3a95: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:30:11.285626 27873 ts_tablet_manager.cc:1905] T 63fc05ed36b34b11b79efdfb43f2012a P 3df449af48e6491382751ff909f2cafe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:30:11.288827 27871 ts_tablet_manager.cc:1918] T 63fc05ed36b34b11b79efdfb43f2012a P 550f6c38624d48d8a73eeaca4d8c3a95: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:30:11.288993 27871 log.cc:1196] T 63fc05ed36b34b11b79efdfb43f2012a P 550f6c38624d48d8a73eeaca4d8c3a95: Deleting WAL directory at /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407783529717-22743-0/minicluster-data/ts-2/wal/wals/63fc05ed36b34b11b79efdfb43f2012a
I20260926 07:30:11.288849 27873 ts_tablet_manager.cc:1918] T 63fc05ed36b34b11b79efdfb43f2012a P 3df449af48e6491382751ff909f2cafe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:30:11.289361 27871 ts_tablet_manager.cc:1939] T 63fc05ed36b34b11b79efdfb43f2012a P 550f6c38624d48d8a73eeaca4d8c3a95: Deleting consensus metadata
I20260926 07:30:11.289390 27873 log.cc:1196] T 63fc05ed36b34b11b79efdfb43f2012a P 3df449af48e6491382751ff909f2cafe: Deleting WAL directory at /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407783529717-22743-0/minicluster-data/ts-1/wal/wals/63fc05ed36b34b11b79efdfb43f2012a
I20260926 07:30:11.289974 27873 ts_tablet_manager.cc:1939] T 63fc05ed36b34b11b79efdfb43f2012a P 3df449af48e6491382751ff909f2cafe: Deleting consensus metadata
I20260926 07:30:11.291116 27368 catalog_manager.cc:4841] TS 550f6c38624d48d8a73eeaca4d8c3a95 (127.22.53.195:37919): tablet 63fc05ed36b34b11b79efdfb43f2012a (table test-table [id=343dd9c8f61e49319dd28f7f630f9e2f]) successfully deleted
I20260926 07:30:11.292089 27365 catalog_manager.cc:4841] TS 3df449af48e6491382751ff909f2cafe (127.22.53.194:45775): tablet 63fc05ed36b34b11b79efdfb43f2012a (table test-table [id=343dd9c8f61e49319dd28f7f630f9e2f]) successfully deleted
I20260926 07:30:11.292181 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 27572
W20260926 07:30:11.292454 27365 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.53.193:41335 (error 108)
I20260926 07:30:11.294595 27842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:11.294251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.193:41335 (local address 127.0.0.1:37950)
0926 07:30:11.294424 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:11.294504 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.193:41335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260926 07:30:11.294974 27365 catalog_manager.cc:4584] TS bb6481aa262643f48d7ae2036ea2fab4 (127.22.53.193:41335): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 63fc05ed36b34b11b79efdfb43f2012a: Network error: Client connection negotiation failed: client connection to 127.22.53.193:41335: connect: Connection refused (error 111)
I20260926 07:30:11.299536 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 27703
I20260926 07:30:11.305768 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 27350
2026-09-26T07:30:11Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1269 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260926 07:30:11.328806 22743 mini_oidc.cc:152] Starting JWKS server
I20260926 07:30:11.330602 22743 webserver.cc:460] Webserver started at https://127.0.0.1:35923/ using document root <none> and password file <none>
I20260926 07:30:11.330833 22743 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 07:30:11.331189 22743 webserver.cc:460] Webserver started at http://127.0.0.1:41963/ using document root <none> and password file <none>
2026-09-26T07:30:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:30:11Z Disabled control of system clock
I20260926 07:30:11.343461 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790407783529717-22743-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790407783529717-22743-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790407783529717-22743-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.53.254:36031
--webserver_interface=127.22.53.254
--webserver_port=0
--builtin_ntp_servers=127.22.53.212:36639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.53.254:36031
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35923/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790407783529717-22743-0/testchainca.cert with env {}
W20260926 07:30:11.463478 27880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:30:11.463833 27880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:30:11.463878 27880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:30:11.468351 27880 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:30:11.468457 27880 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:30:11.468473 27880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:30:11.468492 27880 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:30:11.468510 27880 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 07:30:11.468550 27880 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 07:30:11.468565 27880 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35923/jwks.json
I20260926 07:30:11.474684 27880 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:36639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790407783529717-22743-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790407783529717-22743-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.53.254:36031
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.53.254:36031
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35923/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.53.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790407783529717-22743-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27880
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790407783529717-22743-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:30:11.475992 27880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:30:11.477461 27880 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:30:11.487772 27888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:11.487772 27886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:11.487792 27885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:11.487792 27880 server_base.cc:1034] running on GCE node
I20260926 07:30:11.488843 27880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:30:11.490188 27880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:30:11.491446 27880 hybrid_clock.cc:648] HybridClock initialized: now 1790407811491414 us; error 48 us; skew 500 ppm
I20260926 07:30:11.497671 27880 webserver.cc:460] Webserver started at http://127.22.53.254:44867/ using document root <none> and password file <none>
I20260926 07:30:11.498368 27880 fs_manager.cc:359] Metadata directory not provided
I20260926 07:30:11.498443 27880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:30:11.498649 27880 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:30:11.501235 27880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790407783529717-22743-0/minicluster-data/master-0/data/instance:
uuid: "452447421f174fadb599008aa619b6a4"
format_stamp: "Formatted at 2026-09-26 07:30:11 on dist-test-slave-wdjc"
server_key: "feca99c43e89186937d7faea6e5d98ee"
server_key_iv: "372c86becbd5596ba5d19d366264c62f"
server_key_version: "encryptionkey@0"
I20260926 07:30:11.501813 27880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790407783529717-22743-0/minicluster-data/master-0/wal/instance:
uuid: "452447421f174fadb599008aa619b6a4"
format_stamp: "Formatted at 2026-09-26 07:30:11 on dist-test-slave-wdjc"
server_key: "feca99c43e89186937d7faea6e5d98ee"
server_key_iv: "372c86becbd5596ba5d19d366264c62f"
server_key_version: "encryptionkey@0"
I20260926 07:30:11.506121 27880 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 07:30:11.509203 27894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:11.510878 27880 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:30:11.511065 27880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790407783529717-22743-0/minicluster-data/master-0/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790407783529717-22743-0/minicluster-data/master-0/wal
uuid: "452447421f174fadb599008aa619b6a4"
format_stamp: "Formatted at 2026-09-26 07:30:11 on dist-test-slave-wdjc"
server_key: "feca99c43e89186937d7faea6e5d98ee"
server_key_iv: "372c86becbd5596ba5d19d366264c62f"
server_key_version: "encryptionkey@0"
I20260926 07:30:11.511214 27880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790407783529717-22743-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790407783529717-22743-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790407783529717-22743-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:30:11.523809 27880 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:30:11.524938 27880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:30:11.525200 27880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:30:11.534241 27880 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.254:36031
I20260926 07:30:11.534242 27946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.254:36031 every 8 connection(s)
I20260926 07:30:11.535456 27880 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
I20260926 07:30:11.539240 27947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:30:11.541388 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 27880
I20260926 07:30:11.541543 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790407783529717-22743-0/minicluster-data/master-0/wal/instance
I20260926 07:30:11.541847 22743 external_mini_cluster.cc:1428] Setting key d4e0b3ee14a332431dfdd0c04477b2c4
I20260926 07:30:11.547123 27947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 452447421f174fadb599008aa619b6a4: Bootstrap starting.
I20260926 07:30:11.549952 27947 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 452447421f174fadb599008aa619b6a4: Neither blocks nor log segments found. Creating new log.
I20260926 07:30:11.550864 27947 log.cc:824] T 00000000000000000000000000000000 P 452447421f174fadb599008aa619b6a4: Log is configured to *not* fsync() on all Append() calls
I20260926 07:30:11.553193 27950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:11.543606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39018 (local address 127.22.53.254:36031)
0926 07:30:11.544389 (+ 783us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:11.544406 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:11.544449 (+ 43us) server_negotiation.cc:407] Connection header received
0926 07:30:11.545302 (+ 853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:11.545323 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:11.545801 (+ 478us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:11.546214 (+ 413us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:11.546712 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:11.547877 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:11.549024 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:11.549585 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:11.549826 (+ 241us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:11.550782 (+ 956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:11.550788 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:11.550805 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:30:11.551346 (+ 541us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:11.551431 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:30:11.551463 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:11.551577 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:11.551771 (+ 194us) server_negotiation.cc:299] Negotiation successful
0926 07:30:11.552108 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":160,"threads_started":1}
I20260926 07:30:11.553675 27947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 452447421f174fadb599008aa619b6a4: No bootstrap required, opened a new log
I20260926 07:30:11.556702 22743 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:30:11.558326 27947 raft_consensus.cc:348] T 00000000000000000000000000000000 P 452447421f174fadb599008aa619b6a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "452447421f174fadb599008aa619b6a4" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 36031 } }
I20260926 07:30:11.558718 27947 raft_consensus.cc:374] T 00000000000000000000000000000000 P 452447421f174fadb599008aa619b6a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:30:11.558815 27947 raft_consensus.cc:729] T 00000000000000000000000000000000 P 452447421f174fadb599008aa619b6a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 452447421f174fadb599008aa619b6a4, State: Initialized, Role: FOLLOWER
I20260926 07:30:11.559345 27947 consensus_queue.cc:260] T 00000000000000000000000000000000 P 452447421f174fadb599008aa619b6a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "452447421f174fadb599008aa619b6a4" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 36031 } }
I20260926 07:30:11.559525 27947 raft_consensus.cc:388] T 00000000000000000000000000000000 P 452447421f174fadb599008aa619b6a4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:30:11.559619 27947 raft_consensus.cc:482] T 00000000000000000000000000000000 P 452447421f174fadb599008aa619b6a4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:30:11.559731 27947 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 452447421f174fadb599008aa619b6a4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:30:11.560900 27947 raft_consensus.cc:504] T 00000000000000000000000000000000 P 452447421f174fadb599008aa619b6a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "452447421f174fadb599008aa619b6a4" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 36031 } }
I20260926 07:30:11.561314 27947 leader_election.cc:302] T 00000000000000000000000000000000 P 452447421f174fadb599008aa619b6a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 452447421f174fadb599008aa619b6a4; no voters:
I20260926 07:30:11.561693 27947 leader_election.cc:288] T 00000000000000000000000000000000 P 452447421f174fadb599008aa619b6a4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:30:11.561941 27952 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 452447421f174fadb599008aa619b6a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:30:11.562222 27952 raft_consensus.cc:686] T 00000000000000000000000000000000 P 452447421f174fadb599008aa619b6a4 [term 1 LEADER]: Becoming Leader. State: Replica: 452447421f174fadb599008aa619b6a4, State: Running, Role: LEADER
I20260926 07:30:11.562543 27952 consensus_queue.cc:237] T 00000000000000000000000000000000 P 452447421f174fadb599008aa619b6a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "452447421f174fadb599008aa619b6a4" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 36031 } }
I20260926 07:30:11.563335 27947 sys_catalog.cc:564] T 00000000000000000000000000000000 P 452447421f174fadb599008aa619b6a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:30:11.564430 27953 sys_catalog.cc:455] T 00000000000000000000000000000000 P 452447421f174fadb599008aa619b6a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "452447421f174fadb599008aa619b6a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "452447421f174fadb599008aa619b6a4" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 36031 } } }
I20260926 07:30:11.564634 27953 sys_catalog.cc:458] T 00000000000000000000000000000000 P 452447421f174fadb599008aa619b6a4 [sys.catalog]: This master's current role is: LEADER
I20260926 07:30:11.564920 27954 sys_catalog.cc:455] T 00000000000000000000000000000000 P 452447421f174fadb599008aa619b6a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 452447421f174fadb599008aa619b6a4. Latest consensus state: current_term: 1 leader_uuid: "452447421f174fadb599008aa619b6a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "452447421f174fadb599008aa619b6a4" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 36031 } } }
I20260926 07:30:11.565038 27954 sys_catalog.cc:458] T 00000000000000000000000000000000 P 452447421f174fadb599008aa619b6a4 [sys.catalog]: This master's current role is: LEADER
I20260926 07:30:11.565340 27961 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:30:11.568957 27961 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:30:11.575608 27961 catalog_manager.cc:1347] Generated new cluster ID: c9b946add164441995d51d1e944bd099
I20260926 07:30:11.575717 27961 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:30:11.589510 27961 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:30:11.591038 27961 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:30:11.599349 27961 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 452447421f174fadb599008aa619b6a4: Generated new TSK 0
I20260926 07:30:11.600569 27961 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 07:30:11.640739 27977 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260926 07:30:11.640887 27977 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 07:30:11.641469 27950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:11.623063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39032 (local address 127.22.53.254:36031)
0926 07:30:11.623233 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:11.623238 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:11.623448 (+ 210us) server_negotiation.cc:407] Connection header received
0926 07:30:11.623676 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:11.623680 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:11.623751 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:11.623893 (+ 142us) server_negotiation.cc:226] Negotiated authn=JWT
0926 07:30:11.624196 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:11.625100 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:11.626108 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:11.626489 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:11.626601 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:11.627038 (+ 437us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 07:30:11.641006 (+ 13968us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35923/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 07:30:11.641318 (+ 312us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39032: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35923/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":63}
W20260926 07:30:11.641571 27950 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39032: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35923/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260926 07:30:11.643049 27976 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:30:11.622898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:36031 (local address 127.0.0.1:39032)
0926 07:30:11.623349 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:11.623363 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:11.623493 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:11.623944 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:11.623947 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:11.623954 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0926 07:30:11.624074 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:11.624079 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:11.625271 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:11.625275 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:11.625972 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:11.625981 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:11.626638 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:11.626641 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:11.626721 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:11.626780 (+ 59us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 07:30:11.641270 (+ 14490us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35923/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 07:30:11.641519 (+ 249us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.22.53.254:36031: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35923/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":137,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.22.53.254:36031: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35923/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260926 07:30:11.644192 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 27880
2026-09-26T07:30:11Z chronyd exiting
I20260926 07:30:11.682392 22743 test_util.cc:175] -----------------------------------------------
I20260926 07:30:11.682490 22743 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790407783529717-22743-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (355 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260926 07:30:11.683624 22743 mini_oidc.cc:152] Starting JWKS server
I20260926 07:30:11.684494 22743 webserver.cc:460] Webserver started at https://127.0.0.1:37171/ using document root <none> and password file <none>
I20260926 07:30:11.684711 22743 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 07:30:11.684991 22743 webserver.cc:460] Webserver started at http://127.0.0.1:37519/ using document root <none> and password file <none>
2026-09-26T07:30:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:30:11Z Disabled control of system clock
I20260926 07:30:11.695768 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790407783529717-22743-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790407783529717-22743-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790407783529717-22743-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.53.254:44449
--webserver_interface=127.22.53.254
--webserver_port=0
--builtin_ntp_servers=127.22.53.212:33933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.53.254:44449
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37171/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790407783529717-22743-0/testchainca.cert with env {}
W20260926 07:30:11.818953 27983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:30:11.819286 27983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:30:11.819340 27983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:30:11.823722 27983 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:30:11.823814 27983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:30:11.823830 27983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:30:11.823848 27983 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:30:11.823863 27983 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 07:30:11.823879 27983 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 07:30:11.823892 27983 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37171/jwks.json
I20260926 07:30:11.831666 27983 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:33933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790407783529717-22743-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790407783529717-22743-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.53.254:44449
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.53.254:44449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37171/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.53.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790407783529717-22743-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27983
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790407783529717-22743-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:30:11.833549 27983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:30:11.835232 27983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:30:11.845003 27991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:11.845072 27983 server_base.cc:1034] running on GCE node
W20260926 07:30:11.845093 27988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:11.845003 27989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:11.845902 27983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:30:11.847240 27983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:30:11.848485 27983 hybrid_clock.cc:648] HybridClock initialized: now 1790407811848456 us; error 66 us; skew 500 ppm
I20260926 07:30:11.850839 27983 webserver.cc:460] Webserver started at http://127.22.53.254:33393/ using document root <none> and password file <none>
I20260926 07:30:11.851641 27983 fs_manager.cc:359] Metadata directory not provided
I20260926 07:30:11.851766 27983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:30:11.852046 27983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:30:11.854480 27983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790407783529717-22743-0/minicluster-data/master-0/data/instance:
uuid: "cd5fc8a677f74bf2b068007a2a463cc2"
format_stamp: "Formatted at 2026-09-26 07:30:11 on dist-test-slave-wdjc"
server_key: "62922b7281f85b0ceb82a49455e19ecc"
server_key_iv: "f96d4263a713ceea054d608de5c2b57b"
server_key_version: "encryptionkey@0"
I20260926 07:30:11.855161 27983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790407783529717-22743-0/minicluster-data/master-0/wal/instance:
uuid: "cd5fc8a677f74bf2b068007a2a463cc2"
format_stamp: "Formatted at 2026-09-26 07:30:11 on dist-test-slave-wdjc"
server_key: "62922b7281f85b0ceb82a49455e19ecc"
server_key_iv: "f96d4263a713ceea054d608de5c2b57b"
server_key_version: "encryptionkey@0"
I20260926 07:30:11.859983 27983 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260926 07:30:11.864921 27997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:11.866842 27983 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260926 07:30:11.867090 27983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790407783529717-22743-0/minicluster-data/master-0/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790407783529717-22743-0/minicluster-data/master-0/wal
uuid: "cd5fc8a677f74bf2b068007a2a463cc2"
format_stamp: "Formatted at 2026-09-26 07:30:11 on dist-test-slave-wdjc"
server_key: "62922b7281f85b0ceb82a49455e19ecc"
server_key_iv: "f96d4263a713ceea054d608de5c2b57b"
server_key_version: "encryptionkey@0"
I20260926 07:30:11.867259 27983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790407783529717-22743-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790407783529717-22743-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790407783529717-22743-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:30:11.886651 27983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:30:11.887499 27983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:30:11.887725 27983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:30:11.895737 27983 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.254:44449
I20260926 07:30:11.895768 28049 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.254:44449 every 8 connection(s)
I20260926 07:30:11.896934 27983 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
I20260926 07:30:11.900511 28050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:30:11.903780 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 27983
I20260926 07:30:11.903898 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790407783529717-22743-0/minicluster-data/master-0/wal/instance
I20260926 07:30:11.904256 22743 external_mini_cluster.cc:1428] Setting key 48b80158abd27126c1a88ebe7fcbb4e6
I20260926 07:30:11.907619 28050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd5fc8a677f74bf2b068007a2a463cc2: Bootstrap starting.
I20260926 07:30:11.910266 28050 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd5fc8a677f74bf2b068007a2a463cc2: Neither blocks nor log segments found. Creating new log.
I20260926 07:30:11.911164 28050 log.cc:824] T 00000000000000000000000000000000 P cd5fc8a677f74bf2b068007a2a463cc2: Log is configured to *not* fsync() on all Append() calls
I20260926 07:30:11.913596 28050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd5fc8a677f74bf2b068007a2a463cc2: No bootstrap required, opened a new log
I20260926 07:30:11.914886 28053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:11.905850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47180 (local address 127.22.53.254:44449)
0926 07:30:11.906656 (+ 806us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:11.906672 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:11.906710 (+ 38us) server_negotiation.cc:407] Connection header received
0926 07:30:11.907404 (+ 694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:11.907423 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:11.907961 (+ 538us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:11.908337 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:11.908873 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:11.909916 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:11.911014 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:11.911630 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:11.911838 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:11.912877 (+ 1039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:11.912883 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:11.912900 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:30:11.913358 (+ 458us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:11.913429 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:30:11.913452 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:11.913593 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:11.913725 (+ 132us) server_negotiation.cc:299] Negotiation successful
0926 07:30:11.914048 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":170,"threads_started":1}
I20260926 07:30:11.918874 28050 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd5fc8a677f74bf2b068007a2a463cc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd5fc8a677f74bf2b068007a2a463cc2" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 44449 } }
I20260926 07:30:11.919309 28050 raft_consensus.cc:374] T 00000000000000000000000000000000 P cd5fc8a677f74bf2b068007a2a463cc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:30:11.919294 22743 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:30:11.919402 28050 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd5fc8a677f74bf2b068007a2a463cc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd5fc8a677f74bf2b068007a2a463cc2, State: Initialized, Role: FOLLOWER
I20260926 07:30:11.919930 28050 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd5fc8a677f74bf2b068007a2a463cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd5fc8a677f74bf2b068007a2a463cc2" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 44449 } }
I20260926 07:30:11.920126 28050 raft_consensus.cc:388] T 00000000000000000000000000000000 P cd5fc8a677f74bf2b068007a2a463cc2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:30:11.920221 28050 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd5fc8a677f74bf2b068007a2a463cc2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:30:11.920336 28050 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd5fc8a677f74bf2b068007a2a463cc2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:30:11.921531 28050 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd5fc8a677f74bf2b068007a2a463cc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd5fc8a677f74bf2b068007a2a463cc2" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 44449 } }
I20260926 07:30:11.921954 28050 leader_election.cc:302] T 00000000000000000000000000000000 P cd5fc8a677f74bf2b068007a2a463cc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cd5fc8a677f74bf2b068007a2a463cc2; no voters:
I20260926 07:30:11.922307 28050 leader_election.cc:288] T 00000000000000000000000000000000 P cd5fc8a677f74bf2b068007a2a463cc2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:30:11.922453 28055 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd5fc8a677f74bf2b068007a2a463cc2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:30:11.922695 28055 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd5fc8a677f74bf2b068007a2a463cc2 [term 1 LEADER]: Becoming Leader. State: Replica: cd5fc8a677f74bf2b068007a2a463cc2, State: Running, Role: LEADER
I20260926 07:30:11.923050 28055 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd5fc8a677f74bf2b068007a2a463cc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd5fc8a677f74bf2b068007a2a463cc2" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 44449 } }
I20260926 07:30:11.923678 28050 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd5fc8a677f74bf2b068007a2a463cc2 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:30:11.924873 28057 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd5fc8a677f74bf2b068007a2a463cc2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd5fc8a677f74bf2b068007a2a463cc2. Latest consensus state: current_term: 1 leader_uuid: "cd5fc8a677f74bf2b068007a2a463cc2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd5fc8a677f74bf2b068007a2a463cc2" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 44449 } } }
I20260926 07:30:11.925024 28057 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd5fc8a677f74bf2b068007a2a463cc2 [sys.catalog]: This master's current role is: LEADER
I20260926 07:30:11.925463 28056 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd5fc8a677f74bf2b068007a2a463cc2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cd5fc8a677f74bf2b068007a2a463cc2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd5fc8a677f74bf2b068007a2a463cc2" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 44449 } } }
I20260926 07:30:11.925601 28056 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd5fc8a677f74bf2b068007a2a463cc2 [sys.catalog]: This master's current role is: LEADER
I20260926 07:30:11.926446 28064 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:30:11.929590 28064 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:30:11.936326 28064 catalog_manager.cc:1347] Generated new cluster ID: aa43bdd2875448d0be54117b38ceff9f
I20260926 07:30:11.936434 28064 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:30:11.964457 28064 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:30:11.965931 28064 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:30:11.979336 28064 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cd5fc8a677f74bf2b068007a2a463cc2: Generated new TSK 0
I20260926 07:30:11.980362 28064 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 07:30:12.015933 28080 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260926 07:30:12.016048 28080 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 07:30:12.016844 28053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:11.999135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47192 (local address 127.22.53.254:44449)
0926 07:30:11.999302 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:11.999308 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:11.999624 (+ 316us) server_negotiation.cc:407] Connection header received
0926 07:30:11.999806 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:11.999810 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:11.999900 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:12.000214 (+ 314us) server_negotiation.cc:226] Negotiated authn=JWT
0926 07:30:12.000603 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:12.001219 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:12.002669 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:12.002934 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:12.003036 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:12.003447 (+ 411us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 07:30:12.016379 (+ 12932us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37171/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 07:30:12.016716 (+ 337us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47192: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37171/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":65}
W20260926 07:30:12.016938 28053 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47192: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37171/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260926 07:30:12.017053 28079 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:30:11.999048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:44449 (local address 127.0.0.1:47192)
0926 07:30:11.999422 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:11.999438 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:11.999591 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:12.000206 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:12.000209 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:12.000219 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0926 07:30:12.000364 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:12.000371 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:12.001380 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:12.001384 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:12.002495 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:12.002508 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:12.003084 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:12.003089 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:12.003194 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:12.003232 (+ 38us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 07:30:12.016652 (+ 13420us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37171/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 07:30:12.016864 (+ 212us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.22.53.254:44449: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37171/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":146,"threads_started":1}
I20260926 07:30:12.017794 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 27983
2026-09-26T07:30:12Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (374 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260926 07:30:12.058501 22743 mini_oidc.cc:152] Starting JWKS server
I20260926 07:30:12.059442 22743 webserver.cc:460] Webserver started at https://127.0.0.1:35745/ using document root <none> and password file <none>
I20260926 07:30:12.059558 22743 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 07:30:12.059839 22743 webserver.cc:460] Webserver started at http://127.0.0.1:35839/ using document root <none> and password file <none>
2026-09-26T07:30:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:30:12Z Disabled control of system clock
I20260926 07:30:12.069542 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790407783529717-22743-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790407783529717-22743-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790407783529717-22743-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.53.254:42747
--webserver_interface=127.22.53.254
--webserver_port=0
--builtin_ntp_servers=127.22.53.212:43023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.53.254:42747
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35745/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 07:30:12.187059 28086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:30:12.187441 28086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:30:12.187529 28086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:30:12.191821 28086 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:30:12.191954 28086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:30:12.191984 28086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:30:12.192022 28086 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:30:12.192057 28086 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 07:30:12.192091 28086 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 07:30:12.192122 28086 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35745/jwks.json
I20260926 07:30:12.197309 28086 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:43023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790407783529717-22743-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790407783529717-22743-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.53.254:42747
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.53.254:42747
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35745/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.53.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28086
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790407783529717-22743-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:30:12.198572 28086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:30:12.200106 28086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 07:30:12.209754 28086 server_base.cc:1034] running on GCE node
W20260926 07:30:12.209797 28092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:12.209957 28094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:12.209797 28091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:12.210579 28086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:30:12.211812 28086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:30:12.213013 28086 hybrid_clock.cc:648] HybridClock initialized: now 1790407812212982 us; error 50 us; skew 500 ppm
I20260926 07:30:12.215009 28086 webserver.cc:460] Webserver started at http://127.22.53.254:35809/ using document root <none> and password file <none>
I20260926 07:30:12.215641 28086 fs_manager.cc:359] Metadata directory not provided
I20260926 07:30:12.215741 28086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:30:12.215989 28086 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:30:12.218271 28086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790407783529717-22743-0/minicluster-data/master-0/data/instance:
uuid: "afa661c1c35b4c3bb7e45a675d2ecda3"
format_stamp: "Formatted at 2026-09-26 07:30:12 on dist-test-slave-wdjc"
server_key: "305117b73a44a7a03e197c27a2a1c075"
server_key_iv: "8f5d5540ba5166a771032e3bdf52c910"
server_key_version: "encryptionkey@0"
I20260926 07:30:12.218964 28086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790407783529717-22743-0/minicluster-data/master-0/wal/instance:
uuid: "afa661c1c35b4c3bb7e45a675d2ecda3"
format_stamp: "Formatted at 2026-09-26 07:30:12 on dist-test-slave-wdjc"
server_key: "305117b73a44a7a03e197c27a2a1c075"
server_key_iv: "8f5d5540ba5166a771032e3bdf52c910"
server_key_version: "encryptionkey@0"
I20260926 07:30:12.223675 28086 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260926 07:30:12.228811 28100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:12.231009 28086 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.002s
I20260926 07:30:12.231197 28086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790407783529717-22743-0/minicluster-data/master-0/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790407783529717-22743-0/minicluster-data/master-0/wal
uuid: "afa661c1c35b4c3bb7e45a675d2ecda3"
format_stamp: "Formatted at 2026-09-26 07:30:12 on dist-test-slave-wdjc"
server_key: "305117b73a44a7a03e197c27a2a1c075"
server_key_iv: "8f5d5540ba5166a771032e3bdf52c910"
server_key_version: "encryptionkey@0"
I20260926 07:30:12.231384 28086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790407783529717-22743-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790407783529717-22743-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790407783529717-22743-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:30:12.247298 28086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:30:12.248100 28086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:30:12.248317 28086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:30:12.255668 28086 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.254:42747
I20260926 07:30:12.255672 28152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.254:42747 every 8 connection(s)
I20260926 07:30:12.256858 28086 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
I20260926 07:30:12.260337 28153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:30:12.266319 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 28086
I20260926 07:30:12.266445 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790407783529717-22743-0/minicluster-data/master-0/wal/instance
I20260926 07:30:12.266624 28153 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afa661c1c35b4c3bb7e45a675d2ecda3: Bootstrap starting.
I20260926 07:30:12.266772 22743 external_mini_cluster.cc:1428] Setting key 1a7b3d9d106e8d8a1433560d888bea5f
I20260926 07:30:12.269761 28153 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P afa661c1c35b4c3bb7e45a675d2ecda3: Neither blocks nor log segments found. Creating new log.
I20260926 07:30:12.270800 28153 log.cc:824] T 00000000000000000000000000000000 P afa661c1c35b4c3bb7e45a675d2ecda3: Log is configured to *not* fsync() on all Append() calls
I20260926 07:30:12.273583 28153 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afa661c1c35b4c3bb7e45a675d2ecda3: No bootstrap required, opened a new log
I20260926 07:30:12.278751 28156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:12.268405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38200 (local address 127.22.53.254:42747)
0926 07:30:12.269252 (+ 847us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:12.269271 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:12.269317 (+ 46us) server_negotiation.cc:407] Connection header received
0926 07:30:12.270352 (+ 1035us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:12.270378 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:12.270963 (+ 585us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:12.271380 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:12.271784 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:12.273299 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:12.274366 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:12.275098 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:12.275322 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:12.276239 (+ 917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:12.276247 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:12.276268 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:30:12.276890 (+ 622us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:12.276968 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:30:12.276996 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:12.277109 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:12.277296 (+ 187us) server_negotiation.cc:299] Negotiation successful
0926 07:30:12.277698 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":196,"threads_started":1}
I20260926 07:30:12.278623 28153 raft_consensus.cc:348] T 00000000000000000000000000000000 P afa661c1c35b4c3bb7e45a675d2ecda3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afa661c1c35b4c3bb7e45a675d2ecda3" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 42747 } }
I20260926 07:30:12.278910 28153 raft_consensus.cc:374] T 00000000000000000000000000000000 P afa661c1c35b4c3bb7e45a675d2ecda3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:30:12.278970 28153 raft_consensus.cc:729] T 00000000000000000000000000000000 P afa661c1c35b4c3bb7e45a675d2ecda3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afa661c1c35b4c3bb7e45a675d2ecda3, State: Initialized, Role: FOLLOWER
I20260926 07:30:12.279402 28153 consensus_queue.cc:260] T 00000000000000000000000000000000 P afa661c1c35b4c3bb7e45a675d2ecda3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afa661c1c35b4c3bb7e45a675d2ecda3" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 42747 } }
I20260926 07:30:12.279521 28153 raft_consensus.cc:388] T 00000000000000000000000000000000 P afa661c1c35b4c3bb7e45a675d2ecda3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:30:12.279560 28153 raft_consensus.cc:482] T 00000000000000000000000000000000 P afa661c1c35b4c3bb7e45a675d2ecda3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:30:12.279627 28153 raft_consensus.cc:3037] T 00000000000000000000000000000000 P afa661c1c35b4c3bb7e45a675d2ecda3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:30:12.280747 28153 raft_consensus.cc:504] T 00000000000000000000000000000000 P afa661c1c35b4c3bb7e45a675d2ecda3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afa661c1c35b4c3bb7e45a675d2ecda3" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 42747 } }
I20260926 07:30:12.281169 28153 leader_election.cc:302] T 00000000000000000000000000000000 P afa661c1c35b4c3bb7e45a675d2ecda3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: afa661c1c35b4c3bb7e45a675d2ecda3; no voters:
I20260926 07:30:12.281550 28153 leader_election.cc:288] T 00000000000000000000000000000000 P afa661c1c35b4c3bb7e45a675d2ecda3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:30:12.281690 28158 raft_consensus.cc:2781] T 00000000000000000000000000000000 P afa661c1c35b4c3bb7e45a675d2ecda3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:30:12.281935 28158 raft_consensus.cc:686] T 00000000000000000000000000000000 P afa661c1c35b4c3bb7e45a675d2ecda3 [term 1 LEADER]: Becoming Leader. State: Replica: afa661c1c35b4c3bb7e45a675d2ecda3, State: Running, Role: LEADER
I20260926 07:30:12.282029 22743 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:30:12.282258 28158 consensus_queue.cc:237] T 00000000000000000000000000000000 P afa661c1c35b4c3bb7e45a675d2ecda3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afa661c1c35b4c3bb7e45a675d2ecda3" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 42747 } }
I20260926 07:30:12.283017 28153 sys_catalog.cc:564] T 00000000000000000000000000000000 P afa661c1c35b4c3bb7e45a675d2ecda3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:30:12.283910 28159 sys_catalog.cc:455] T 00000000000000000000000000000000 P afa661c1c35b4c3bb7e45a675d2ecda3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "afa661c1c35b4c3bb7e45a675d2ecda3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afa661c1c35b4c3bb7e45a675d2ecda3" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 42747 } } }
I20260926 07:30:12.284044 28159 sys_catalog.cc:458] T 00000000000000000000000000000000 P afa661c1c35b4c3bb7e45a675d2ecda3 [sys.catalog]: This master's current role is: LEADER
I20260926 07:30:12.284329 28160 sys_catalog.cc:455] T 00000000000000000000000000000000 P afa661c1c35b4c3bb7e45a675d2ecda3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader afa661c1c35b4c3bb7e45a675d2ecda3. Latest consensus state: current_term: 1 leader_uuid: "afa661c1c35b4c3bb7e45a675d2ecda3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afa661c1c35b4c3bb7e45a675d2ecda3" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 42747 } } }
I20260926 07:30:12.284420 28160 sys_catalog.cc:458] T 00000000000000000000000000000000 P afa661c1c35b4c3bb7e45a675d2ecda3 [sys.catalog]: This master's current role is: LEADER
I20260926 07:30:12.284852 28167 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:30:12.288890 28167 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:30:12.295626 28167 catalog_manager.cc:1347] Generated new cluster ID: da12139b7e634e26be46301de15c0c8c
I20260926 07:30:12.295729 28167 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:30:12.313694 28167 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:30:12.314766 28167 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:30:12.336876 28167 catalog_manager.cc:5868] T 00000000000000000000000000000000 P afa661c1c35b4c3bb7e45a675d2ecda3: Generated new TSK 0
I20260926 07:30:12.337738 28167 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:30:12.356681 28156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:12.351314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38206 (local address 127.22.53.254:42747)
0926 07:30:12.351474 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:12.351479 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:12.351616 (+ 137us) server_negotiation.cc:407] Connection header received
0926 07:30:12.351798 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:12.351801 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:12.351853 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:12.351953 (+ 100us) server_negotiation.cc:226] Negotiated authn=JWT
0926 07:30:12.352237 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:12.352848 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:12.354169 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:12.354400 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:12.354502 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:12.354847 (+ 345us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 07:30:12.356190 (+ 1343us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35745/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 07:30:12.356482 (+ 292us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38206: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35745/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":60}
W20260926 07:30:12.356776 28156 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38206: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35745/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260926 07:30:12.359578 28182 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:30:12.351126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:42747 (local address 127.0.0.1:38206)
0926 07:30:12.351510 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:12.351528 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:12.351655 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:12.351984 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:12.351987 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:12.351996 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0926 07:30:12.352122 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:12.352129 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:12.352998 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:12.353002 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:12.354032 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:12.354044 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:12.354537 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:12.354541 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:12.354632 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:12.354665 (+ 33us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 07:30:12.356362 (+ 1697us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35745/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 07:30:12.359391 (+ 3029us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.22.53.254:42747: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35745/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":280,"mutex_wait_us":2772,"thread_start_us":135,"threads_started":1}
I20260926 07:30:12.360246 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 28086
E20260926 07:30:12.370887 28183 webserver.cc:533] Webserver: sslize failed:
E20260926 07:30:12.370982 28183 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-26T07:30:12Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (343 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-26T07:30:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:30:12Z Disabled control of system clock
I20260926 07:30:12.412503 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790407783529717-22743-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790407783529717-22743-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790407783529717-22743-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.53.254:35771
--webserver_interface=127.22.53.254
--webserver_port=0
--builtin_ntp_servers=127.22.53.212:33539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.53.254:35771
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790407783529717-22743-0/testchainca.cert with env {}
W20260926 07:30:12.537477 28187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:30:12.537879 28187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:30:12.537957 28187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:30:12.542497 28187 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:30:12.542621 28187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:30:12.542652 28187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:30:12.542697 28187 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:30:12.542735 28187 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 07:30:12.542800 28187 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 07:30:12.542835 28187 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260926 07:30:12.548971 28187 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:33539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790407783529717-22743-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790407783529717-22743-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.53.254:35771
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.53.254:35771
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.53.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790407783529717-22743-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28187
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790407783529717-22743-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:30:12.550217 28187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:30:12.552104 28187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:30:12.562222 28195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:12.562467 28187 server_base.cc:1034] running on GCE node
W20260926 07:30:12.562330 28192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:12.562325 28193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:12.563388 28187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:30:12.564795 28187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:30:12.566027 28187 hybrid_clock.cc:648] HybridClock initialized: now 1790407812566016 us; error 34 us; skew 500 ppm
I20260926 07:30:12.568279 28187 webserver.cc:460] Webserver started at http://127.22.53.254:46837/ using document root <none> and password file <none>
I20260926 07:30:12.569028 28187 fs_manager.cc:359] Metadata directory not provided
I20260926 07:30:12.569108 28187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:30:12.569392 28187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:30:12.571928 28187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790407783529717-22743-0/minicluster-data/master-0/data/instance:
uuid: "7b2f38a6814a41c0bb6a8ae5a53b008a"
format_stamp: "Formatted at 2026-09-26 07:30:12 on dist-test-slave-wdjc"
server_key: "be3aacb513b8eb6d49005bbc5a35295c"
server_key_iv: "5de70fc1b5397804244dfe41ce513df7"
server_key_version: "encryptionkey@0"
I20260926 07:30:12.572713 28187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790407783529717-22743-0/minicluster-data/master-0/wal/instance:
uuid: "7b2f38a6814a41c0bb6a8ae5a53b008a"
format_stamp: "Formatted at 2026-09-26 07:30:12 on dist-test-slave-wdjc"
server_key: "be3aacb513b8eb6d49005bbc5a35295c"
server_key_iv: "5de70fc1b5397804244dfe41ce513df7"
server_key_version: "encryptionkey@0"
I20260926 07:30:12.577647 28187 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260926 07:30:12.581180 28201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:12.583231 28187 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260926 07:30:12.583551 28187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790407783529717-22743-0/minicluster-data/master-0/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790407783529717-22743-0/minicluster-data/master-0/wal
uuid: "7b2f38a6814a41c0bb6a8ae5a53b008a"
format_stamp: "Formatted at 2026-09-26 07:30:12 on dist-test-slave-wdjc"
server_key: "be3aacb513b8eb6d49005bbc5a35295c"
server_key_iv: "5de70fc1b5397804244dfe41ce513df7"
server_key_version: "encryptionkey@0"
I20260926 07:30:12.583946 28187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790407783529717-22743-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790407783529717-22743-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790407783529717-22743-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:30:12.601037 28187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:30:12.601877 28187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:30:12.602121 28187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:30:12.611970 28187 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.254:35771
I20260926 07:30:12.611999 28253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.254:35771 every 8 connection(s)
I20260926 07:30:12.613158 28187 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
I20260926 07:30:12.617506 28254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:30:12.620692 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 28187
I20260926 07:30:12.620855 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790407783529717-22743-0/minicluster-data/master-0/wal/instance
I20260926 07:30:12.621135 22743 external_mini_cluster.cc:1428] Setting key 9410869f3992c147632a7196701f0376
I20260926 07:30:12.625171 28254 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b2f38a6814a41c0bb6a8ae5a53b008a: Bootstrap starting.
I20260926 07:30:12.628952 28254 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b2f38a6814a41c0bb6a8ae5a53b008a: Neither blocks nor log segments found. Creating new log.
I20260926 07:30:12.630296 28254 log.cc:824] T 00000000000000000000000000000000 P 7b2f38a6814a41c0bb6a8ae5a53b008a: Log is configured to *not* fsync() on all Append() calls
I20260926 07:30:12.632752 28254 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b2f38a6814a41c0bb6a8ae5a53b008a: No bootstrap required, opened a new log
I20260926 07:30:12.633834 28257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:12.623645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42536 (local address 127.22.53.254:35771)
0926 07:30:12.624959 (+ 1314us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:12.624983 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:12.625033 (+ 50us) server_negotiation.cc:407] Connection header received
0926 07:30:12.625872 (+ 839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:12.625896 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:12.626414 (+ 518us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:12.626786 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:12.627312 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:12.628466 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:12.630062 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:12.630731 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:12.630960 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:12.631873 (+ 913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:12.631878 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:12.631894 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:30:12.632318 (+ 424us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:12.632382 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:30:12.632404 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:12.632521 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:12.632712 (+ 191us) server_negotiation.cc:299] Negotiation successful
0926 07:30:12.633032 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":182,"threads_started":1}
I20260926 07:30:12.636365 28254 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b2f38a6814a41c0bb6a8ae5a53b008a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b2f38a6814a41c0bb6a8ae5a53b008a" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 35771 } }
I20260926 07:30:12.636745 28254 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7b2f38a6814a41c0bb6a8ae5a53b008a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:30:12.636792 28254 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b2f38a6814a41c0bb6a8ae5a53b008a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b2f38a6814a41c0bb6a8ae5a53b008a, State: Initialized, Role: FOLLOWER
I20260926 07:30:12.637356 28254 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b2f38a6814a41c0bb6a8ae5a53b008a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b2f38a6814a41c0bb6a8ae5a53b008a" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 35771 } }
I20260926 07:30:12.637532 28254 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7b2f38a6814a41c0bb6a8ae5a53b008a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:30:12.637635 28254 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b2f38a6814a41c0bb6a8ae5a53b008a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:30:12.637760 28254 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b2f38a6814a41c0bb6a8ae5a53b008a [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:30:12.637944 22743 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:30:12.638917 28254 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b2f38a6814a41c0bb6a8ae5a53b008a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b2f38a6814a41c0bb6a8ae5a53b008a" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 35771 } }
I20260926 07:30:12.639343 28254 leader_election.cc:302] T 00000000000000000000000000000000 P 7b2f38a6814a41c0bb6a8ae5a53b008a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b2f38a6814a41c0bb6a8ae5a53b008a; no voters:
I20260926 07:30:12.639736 28254 leader_election.cc:288] T 00000000000000000000000000000000 P 7b2f38a6814a41c0bb6a8ae5a53b008a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:30:12.639827 28259 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7b2f38a6814a41c0bb6a8ae5a53b008a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:30:12.640153 28259 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7b2f38a6814a41c0bb6a8ae5a53b008a [term 1 LEADER]: Becoming Leader. State: Replica: 7b2f38a6814a41c0bb6a8ae5a53b008a, State: Running, Role: LEADER
I20260926 07:30:12.640491 28259 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7b2f38a6814a41c0bb6a8ae5a53b008a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b2f38a6814a41c0bb6a8ae5a53b008a" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 35771 } }
I20260926 07:30:12.641145 28254 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b2f38a6814a41c0bb6a8ae5a53b008a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:30:12.642527 28261 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b2f38a6814a41c0bb6a8ae5a53b008a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b2f38a6814a41c0bb6a8ae5a53b008a. Latest consensus state: current_term: 1 leader_uuid: "7b2f38a6814a41c0bb6a8ae5a53b008a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b2f38a6814a41c0bb6a8ae5a53b008a" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 35771 } } }
I20260926 07:30:12.642540 28260 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b2f38a6814a41c0bb6a8ae5a53b008a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7b2f38a6814a41c0bb6a8ae5a53b008a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b2f38a6814a41c0bb6a8ae5a53b008a" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 35771 } } }
I20260926 07:30:12.642684 28260 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b2f38a6814a41c0bb6a8ae5a53b008a [sys.catalog]: This master's current role is: LEADER
I20260926 07:30:12.642683 28261 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b2f38a6814a41c0bb6a8ae5a53b008a [sys.catalog]: This master's current role is: LEADER
I20260926 07:30:12.643127 28269 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:30:12.646677 28269 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:30:12.653352 28269 catalog_manager.cc:1347] Generated new cluster ID: bc3bcfbcbb664d3fbc8e3fd10c453cc3
I20260926 07:30:12.653445 28269 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:30:12.673483 28269 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:30:12.674873 28269 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:30:12.683712 28269 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7b2f38a6814a41c0bb6a8ae5a53b008a: Generated new TSK 0
I20260926 07:30:12.684684 28269 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:30:12.781819 28257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:12.698857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42548 (local address 127.22.53.254:35771)
0926 07:30:12.699030 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:12.699034 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:12.699203 (+ 169us) server_negotiation.cc:407] Connection header received
0926 07:30:12.699374 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:12.699376 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:12.699437 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:12.699548 (+ 111us) server_negotiation.cc:226] Negotiated authn=JWT
0926 07:30:12.699846 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:12.700461 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:12.701855 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:12.702098 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:12.702185 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:12.702609 (+ 424us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 07:30:12.781383 (+ 78774us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 07:30:12.781689 (+ 306us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42548: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":76}
W20260926 07:30:12.781935 28257 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42548: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260926 07:30:12.782115 28283 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:30:12.698672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:35771 (local address 127.0.0.1:42548)
0926 07:30:12.699070 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:12.699087 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:12.699224 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:12.699582 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:12.699585 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:12.699594 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0926 07:30:12.699731 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:12.699738 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:12.700642 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:12.700646 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:12.701720 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:12.701733 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:12.702242 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:12.702246 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:12.702340 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:12.702375 (+ 35us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 07:30:12.781673 (+ 79298us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 07:30:12.781907 (+ 234us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.22.53.254:35771: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":136,"threads_started":1}
I20260926 07:30:12.783068 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 28187
2026-09-26T07:30:12Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (402 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (138 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (136 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407783529717-22743-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:30:13 dist-test-slave-wdjc krb5kdc[28295](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:30:13 dist-test-slave-wdjc krb5kdc[28295](info): set up 2 sockets
Sep 26 07:30:13 dist-test-slave-wdjc krb5kdc[28295](info): commencing operation
krb5kdc: starting...
W20260926 07:30:14.137575 22743 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407783529717-22743-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407783529717-22743-0/krb5kdc/test-user.keytab.
Sep 26 07:30:14 dist-test-slave-wdjc krb5kdc[28295](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407814, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:30:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:30:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.53.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.53.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.53.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab.
Entry for principal kudu/127.22.53.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab.
I20260926 07:30:14.277359 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407783529717-22743-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407783529717-22743-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407783529717-22743-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.53.254:41479
--webserver_interface=127.22.53.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab
--principal=kudu/127.22.53.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:44813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.53.254:41479
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:30:14.402902 28308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:30:14.403251 28308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:30:14.403297 28308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:30:14.408119 28308 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:30:14.408216 28308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:30:14.408238 28308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:30:14.408257 28308 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:30:14.408274 28308 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:30:14.414388 28308 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:44813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407783529717-22743-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407783529717-22743-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.53.254:41479
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab
--principal=kudu/127.22.53.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.53.254:41479
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.53.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28308
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407783529717-22743-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:30:14.415693 28308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:30:14.416915 28308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:30:14.426908 28314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:14.426914 28316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:14.426915 28313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:14.427438 28308 server_base.cc:1034] running on GCE node
I20260926 07:30:14.428237 28308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:30:14.430016 28308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:30:14.431335 28308 hybrid_clock.cc:648] HybridClock initialized: now 1790407814431301 us; error 63 us; skew 500 ppm
Sep 26 07:30:14 dist-test-slave-wdjc krb5kdc[28295](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407814, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:30:14.435144 28308 init.cc:377] Logged in from keytab as kudu/127.22.53.254@KRBTEST.COM (short username kudu)
I20260926 07:30:14.436632 28308 webserver.cc:460] Webserver started at http://127.22.53.254:41211/ using document root <none> and password file <none>
I20260926 07:30:14.437387 28308 fs_manager.cc:359] Metadata directory not provided
I20260926 07:30:14.437502 28308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:30:14.437846 28308 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:30:14.440902 28308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407783529717-22743-0/minicluster-data/master-0/data/instance:
uuid: "f667f0018dd94b048b32e934f200e450"
format_stamp: "Formatted at 2026-09-26 07:30:14 on dist-test-slave-wdjc"
server_key: "f9577f35c0bd6c2a67f583d62e751e30"
server_key_iv: "b16e9fc934142e7e661cf2a912f4c590"
server_key_version: "encryptionkey@0"
I20260926 07:30:14.441700 28308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407783529717-22743-0/minicluster-data/master-0/wal/instance:
uuid: "f667f0018dd94b048b32e934f200e450"
format_stamp: "Formatted at 2026-09-26 07:30:14 on dist-test-slave-wdjc"
server_key: "f9577f35c0bd6c2a67f583d62e751e30"
server_key_iv: "b16e9fc934142e7e661cf2a912f4c590"
server_key_version: "encryptionkey@0"
I20260926 07:30:14.446533 28308 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 07:30:14.449776 28323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:14.451419 28308 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260926 07:30:14.451609 28308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407783529717-22743-0/minicluster-data/master-0/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407783529717-22743-0/minicluster-data/master-0/wal
uuid: "f667f0018dd94b048b32e934f200e450"
format_stamp: "Formatted at 2026-09-26 07:30:14 on dist-test-slave-wdjc"
server_key: "f9577f35c0bd6c2a67f583d62e751e30"
server_key_iv: "b16e9fc934142e7e661cf2a912f4c590"
server_key_version: "encryptionkey@0"
I20260926 07:30:14.451751 28308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407783529717-22743-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407783529717-22743-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407783529717-22743-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:30:14.483474 28308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:30:14.487774 28308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:30:14.487998 28308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:30:14.496419 28308 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.254:41479
I20260926 07:30:14.496510 28375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.254:41479 every 8 connection(s)
I20260926 07:30:14.497618 28308 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
I20260926 07:30:14.501179 28376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:30:14.506479 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 28308
I20260926 07:30:14.506606 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407783529717-22743-0/minicluster-data/master-0/wal/instance
I20260926 07:30:14.506992 22743 external_mini_cluster.cc:1428] Setting key d37d551fea9746004ddfa9fc045f341a
I20260926 07:30:14.509639 28376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f667f0018dd94b048b32e934f200e450: Bootstrap starting.
I20260926 07:30:14.512176 28376 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f667f0018dd94b048b32e934f200e450: Neither blocks nor log segments found. Creating new log.
I20260926 07:30:14.513399 28376 log.cc:824] T 00000000000000000000000000000000 P f667f0018dd94b048b32e934f200e450: Log is configured to *not* fsync() on all Append() calls
I20260926 07:30:14.515707 28376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f667f0018dd94b048b32e934f200e450: No bootstrap required, opened a new log
Sep 26 07:30:14 dist-test-slave-wdjc krb5kdc[28295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407814, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.53.254@KRBTEST.COM
I20260926 07:30:14.519567 28376 raft_consensus.cc:348] T 00000000000000000000000000000000 P f667f0018dd94b048b32e934f200e450 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f667f0018dd94b048b32e934f200e450" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 41479 } }
I20260926 07:30:14.519906 28376 raft_consensus.cc:374] T 00000000000000000000000000000000 P f667f0018dd94b048b32e934f200e450 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:30:14.519994 28376 raft_consensus.cc:729] T 00000000000000000000000000000000 P f667f0018dd94b048b32e934f200e450 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f667f0018dd94b048b32e934f200e450, State: Initialized, Role: FOLLOWER
I20260926 07:30:14.520627 28376 consensus_queue.cc:260] T 00000000000000000000000000000000 P f667f0018dd94b048b32e934f200e450 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f667f0018dd94b048b32e934f200e450" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 41479 } }
I20260926 07:30:14.520814 28376 raft_consensus.cc:388] T 00000000000000000000000000000000 P f667f0018dd94b048b32e934f200e450 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:30:14.520915 28376 raft_consensus.cc:482] T 00000000000000000000000000000000 P f667f0018dd94b048b32e934f200e450 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:30:14.521029 28376 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f667f0018dd94b048b32e934f200e450 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:30:14.522141 28376 raft_consensus.cc:504] T 00000000000000000000000000000000 P f667f0018dd94b048b32e934f200e450 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f667f0018dd94b048b32e934f200e450" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 41479 } }
I20260926 07:30:14.522565 28376 leader_election.cc:302] T 00000000000000000000000000000000 P f667f0018dd94b048b32e934f200e450 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f667f0018dd94b048b32e934f200e450; no voters:
I20260926 07:30:14.523010 28376 leader_election.cc:288] T 00000000000000000000000000000000 P f667f0018dd94b048b32e934f200e450 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:30:14.523118 28381 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f667f0018dd94b048b32e934f200e450 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:30:14.523360 28381 raft_consensus.cc:686] T 00000000000000000000000000000000 P f667f0018dd94b048b32e934f200e450 [term 1 LEADER]: Becoming Leader. State: Replica: f667f0018dd94b048b32e934f200e450, State: Running, Role: LEADER
I20260926 07:30:14.523739 28381 consensus_queue.cc:237] T 00000000000000000000000000000000 P f667f0018dd94b048b32e934f200e450 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f667f0018dd94b048b32e934f200e450" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 41479 } }
I20260926 07:30:14.524438 28379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:14.508773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51438 (local address 127.22.53.254:41479)
0926 07:30:14.509607 (+ 834us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:14.509626 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:14.509716 (+ 90us) server_negotiation.cc:407] Connection header received
0926 07:30:14.510531 (+ 815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:14.510553 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:14.510943 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:14.511349 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:14.512249 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:14.513396 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:14.514316 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:14.514837 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:14.515034 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:14.518089 (+ 3055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:14.518095 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:14.518115 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:14.518160 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:14.520849 (+ 2689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:14.521498 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:14.521505 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:14.521520 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:14.521680 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:14.522008 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:14.522012 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:14.522014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:14.522545 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:14.522792 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:14.523125 (+ 333us) server_negotiation.cc:299] Negotiation successful
0926 07:30:14.523498 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":673,"thread_start_us":197,"threads_started":1}
I20260926 07:30:14.524513 28376 sys_catalog.cc:564] T 00000000000000000000000000000000 P f667f0018dd94b048b32e934f200e450 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:30:14.525935 28383 sys_catalog.cc:455] T 00000000000000000000000000000000 P f667f0018dd94b048b32e934f200e450 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f667f0018dd94b048b32e934f200e450. Latest consensus state: current_term: 1 leader_uuid: "f667f0018dd94b048b32e934f200e450" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f667f0018dd94b048b32e934f200e450" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 41479 } } }
I20260926 07:30:14.525966 28382 sys_catalog.cc:455] T 00000000000000000000000000000000 P f667f0018dd94b048b32e934f200e450 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f667f0018dd94b048b32e934f200e450" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f667f0018dd94b048b32e934f200e450" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 41479 } } }
I20260926 07:30:14.526124 28382 sys_catalog.cc:458] T 00000000000000000000000000000000 P f667f0018dd94b048b32e934f200e450 [sys.catalog]: This master's current role is: LEADER
I20260926 07:30:14.526120 28383 sys_catalog.cc:458] T 00000000000000000000000000000000 P f667f0018dd94b048b32e934f200e450 [sys.catalog]: This master's current role is: LEADER
I20260926 07:30:14.527359 28390 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:30:14.531549 28390 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:30:14.539458 28390 catalog_manager.cc:1347] Generated new cluster ID: 0e7a8dbc92b44e7f82aee4ccadb38011
I20260926 07:30:14.539562 28390 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:30:14.550937 28390 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:30:14.552386 28390 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:30:14.565495 28390 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f667f0018dd94b048b32e934f200e450: Generated new TSK 0
I20260926 07:30:14.566533 28390 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.53.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.53.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.53.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.193.keytab.
Entry for principal kudu/127.22.53.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.193.keytab.
I20260926 07:30:14.606504 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407783529717-22743-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407783529717-22743-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407783529717-22743-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407783529717-22743-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.53.193:0
--local_ip_for_outbound_sockets=127.22.53.193
--webserver_interface=127.22.53.193
--webserver_port=0
--tserver_master_addrs=127.22.53.254:41479
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.193.keytab
--principal=kudu/127.22.53.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:44813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:30:14.734408 28402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:30:14.735034 28402 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:30:14.735260 28402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:30:14.741465 28402 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:30:14.741568 28402 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:30:14.741698 28402 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.53.193
I20260926 07:30:14.748191 28402 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:44813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407783529717-22743-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407783529717-22743-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.193.keytab
--principal=kudu/127.22.53.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.53.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407783529717-22743-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.53.193
--webserver_port=0
--tserver_master_addrs=127.22.53.254:41479
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28402
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.53.193
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407783529717-22743-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:30:14.749498 28402 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:30:14.750656 28402 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:30:14.761807 28407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:14.761888 28402 server_base.cc:1034] running on GCE node
W20260926 07:30:14.761818 28408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:14.761818 28410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:14.762583 28402 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:30:14.763351 28402 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:30:14.764597 28402 hybrid_clock.cc:648] HybridClock initialized: now 1790407814764579 us; error 32 us; skew 500 ppm
Sep 26 07:30:14 dist-test-slave-wdjc krb5kdc[28295](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407814, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:30:14.767705 28402 init.cc:377] Logged in from keytab as kudu/127.22.53.193@KRBTEST.COM (short username kudu)
I20260926 07:30:14.768868 28402 webserver.cc:460] Webserver started at http://127.22.53.193:38187/ using document root <none> and password file <none>
I20260926 07:30:14.769524 28402 fs_manager.cc:359] Metadata directory not provided
I20260926 07:30:14.769626 28402 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:30:14.769886 28402 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:30:14.772765 28402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407783529717-22743-0/minicluster-data/ts-0/data/instance:
uuid: "d990b931faef4e8da6f5f26e3e72064f"
format_stamp: "Formatted at 2026-09-26 07:30:14 on dist-test-slave-wdjc"
server_key: "a19542fc231b654bc0569602d3915c1b"
server_key_iv: "c723e4db7f33c0460e67ab63de22c40b"
server_key_version: "encryptionkey@0"
I20260926 07:30:14.773466 28402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407783529717-22743-0/minicluster-data/ts-0/wal/instance:
uuid: "d990b931faef4e8da6f5f26e3e72064f"
format_stamp: "Formatted at 2026-09-26 07:30:14 on dist-test-slave-wdjc"
server_key: "a19542fc231b654bc0569602d3915c1b"
server_key_iv: "c723e4db7f33c0460e67ab63de22c40b"
server_key_version: "encryptionkey@0"
I20260926 07:30:14.778043 28402 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260926 07:30:14.781390 28417 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:14.782992 28402 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 07:30:14.783186 28402 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407783529717-22743-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407783529717-22743-0/minicluster-data/ts-0/wal
uuid: "d990b931faef4e8da6f5f26e3e72064f"
format_stamp: "Formatted at 2026-09-26 07:30:14 on dist-test-slave-wdjc"
server_key: "a19542fc231b654bc0569602d3915c1b"
server_key_iv: "c723e4db7f33c0460e67ab63de22c40b"
server_key_version: "encryptionkey@0"
I20260926 07:30:14.783337 28402 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407783529717-22743-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407783529717-22743-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407783529717-22743-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:30:14.799809 28402 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:30:14.803835 28402 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:30:14.804080 28402 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:30:14.804859 28402 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:30:14.805861 28402 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:30:14.805948 28402 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:14.806023 28402 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:30:14.806068 28402 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:14.816551 28402 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.193:39183
I20260926 07:30:14.816614 28530 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.193:39183 every 8 connection(s)
I20260926 07:30:14.818087 28402 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407783529717-22743-0/minicluster-data/ts-0/data/info.pb
I20260926 07:30:14.825194 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 28402
I20260926 07:30:14.825366 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407783529717-22743-0/minicluster-data/ts-0/wal/instance
I20260926 07:30:14.825738 22743 external_mini_cluster.cc:1428] Setting key 8bbf68d609314f61ea7cbc28f9bb7631
Sep 26 07:30:14 dist-test-slave-wdjc krb5kdc[28295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407814, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.193@KRBTEST.COM for kudu/127.22.53.254@KRBTEST.COM
I20260926 07:30:14.837160 28379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:14.820456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.193:52549 (local address 127.22.53.254:41479)
0926 07:30:14.820667 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:14.820672 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:14.821767 (+ 1095us) server_negotiation.cc:407] Connection header received
0926 07:30:14.823461 (+ 1694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:14.823464 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:14.823519 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:14.823610 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:14.825349 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:14.826031 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:14.827900 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:14.828180 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:14.828275 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:14.831854 (+ 3579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:14.831863 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:14.831870 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:14.831915 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:14.834156 (+ 2241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:14.834945 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:14.834949 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:14.834952 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:14.835070 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:14.835592 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:14.835599 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:14.835604 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:14.835939 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:14.836075 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:14.836743 (+ 668us) server_negotiation.cc:299] Negotiation successful
0926 07:30:14.836952 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
WARNING: no policy specified for kudu/127.22.53.194@KRBTEST.COM; defaulting to no policy
I20260926 07:30:14.838102 28533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:14.820957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:41479 (local address 127.22.53.193:52549)
0926 07:30:14.821534 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:14.821596 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:14.823197 (+ 1601us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:14.823800 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:14.823808 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:14.824253 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:14.825018 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:14.825045 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:14.826255 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:14.826258 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:14.827747 (+ 1489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:14.827763 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:14.828371 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:14.828378 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:14.828684 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:14.829421 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:14.829453 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:14.831583 (+ 2130us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:30:14.834374 (+ 2791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:14.834381 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:14.834397 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:14.834787 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:14.835248 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:14.835251 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:14.835253 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:14.835451 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:14.836072 (+ 621us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:30:14.836079 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:30:14.836388 (+ 309us) client_negotiation.cc:769] Sending connection context
0926 07:30:14.836677 (+ 289us) client_negotiation.cc:240] Negotiation successful
0926 07:30:14.837086 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":174,"threads_started":1}
I20260926 07:30:14.839649 28531 heartbeater.cc:347] Connected to a master server at 127.22.53.254:41479
I20260926 07:30:14.840016 28531 heartbeater.cc:464] Registering TS with master...
I20260926 07:30:14.840758 28531 heartbeater.cc:511] Master 127.22.53.254:41479 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.53.194@KRBTEST.COM" created.
I20260926 07:30:14.842762 28339 ts_manager.cc:194] Registered new tserver with Master: d990b931faef4e8da6f5f26e3e72064f (127.22.53.193:39183)
I20260926 07:30:14.844254 28339 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.53.193@KRBTEST.COM'} at 127.22.53.193:52549
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.53.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.194.keytab.
Entry for principal kudu/127.22.53.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.194.keytab.
I20260926 07:30:14.860390 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407783529717-22743-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407783529717-22743-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407783529717-22743-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407783529717-22743-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.53.194:0
--local_ip_for_outbound_sockets=127.22.53.194
--webserver_interface=127.22.53.194
--webserver_port=0
--tserver_master_addrs=127.22.53.254:41479
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.194.keytab
--principal=kudu/127.22.53.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:44813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:30:14.983697 28536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:30:14.984050 28536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:30:14.984145 28536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:30:14.988869 28536 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:30:14.988956 28536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:30:14.989024 28536 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.53.194
I20260926 07:30:14.995026 28536 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:44813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407783529717-22743-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407783529717-22743-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.194.keytab
--principal=kudu/127.22.53.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.53.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407783529717-22743-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.53.194
--webserver_port=0
--tserver_master_addrs=127.22.53.254:41479
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28536
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.53.194
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407783529717-22743-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:30:14.996296 28536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:30:14.997486 28536 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:30:15.008438 28541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:15.008482 28542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:15.008591 28536 server_base.cc:1034] running on GCE node
W20260926 07:30:15.008478 28544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:15.009276 28536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:30:15.010082 28536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:30:15.011327 28536 hybrid_clock.cc:648] HybridClock initialized: now 1790407815011285 us; error 64 us; skew 500 ppm
Sep 26 07:30:15 dist-test-slave-wdjc krb5kdc[28295](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407815, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:30:15.014777 28536 init.cc:377] Logged in from keytab as kudu/127.22.53.194@KRBTEST.COM (short username kudu)
I20260926 07:30:15.015865 28536 webserver.cc:460] Webserver started at http://127.22.53.194:34375/ using document root <none> and password file <none>
I20260926 07:30:15.016623 28536 fs_manager.cc:359] Metadata directory not provided
I20260926 07:30:15.016733 28536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:30:15.017059 28536 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:30:15.019506 28536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407783529717-22743-0/minicluster-data/ts-1/data/instance:
uuid: "1665934dc5c34c7ca4676464e02f915a"
format_stamp: "Formatted at 2026-09-26 07:30:15 on dist-test-slave-wdjc"
server_key: "3d634d78df1a692453d0eec9b9ea5c2b"
server_key_iv: "858bc80df405b24fb942f5c518a245f3"
server_key_version: "encryptionkey@0"
I20260926 07:30:15.020233 28536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407783529717-22743-0/minicluster-data/ts-1/wal/instance:
uuid: "1665934dc5c34c7ca4676464e02f915a"
format_stamp: "Formatted at 2026-09-26 07:30:15 on dist-test-slave-wdjc"
server_key: "3d634d78df1a692453d0eec9b9ea5c2b"
server_key_iv: "858bc80df405b24fb942f5c518a245f3"
server_key_version: "encryptionkey@0"
I20260926 07:30:15.024976 28536 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 07:30:15.028383 28551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:15.029882 28536 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:30:15.030083 28536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407783529717-22743-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407783529717-22743-0/minicluster-data/ts-1/wal
uuid: "1665934dc5c34c7ca4676464e02f915a"
format_stamp: "Formatted at 2026-09-26 07:30:15 on dist-test-slave-wdjc"
server_key: "3d634d78df1a692453d0eec9b9ea5c2b"
server_key_iv: "858bc80df405b24fb942f5c518a245f3"
server_key_version: "encryptionkey@0"
I20260926 07:30:15.030257 28536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407783529717-22743-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407783529717-22743-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407783529717-22743-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:30:15.057878 28536 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:30:15.061724 28536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:30:15.061968 28536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:30:15.062641 28536 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:30:15.063584 28536 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:30:15.063659 28536 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:15.063741 28536 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:30:15.063788 28536 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:15.074016 28536 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.194:34703
I20260926 07:30:15.074035 28664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.194:34703 every 8 connection(s)
I20260926 07:30:15.075109 28536 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407783529717-22743-0/minicluster-data/ts-1/data/info.pb
I20260926 07:30:15.078137 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 28536
I20260926 07:30:15.078315 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407783529717-22743-0/minicluster-data/ts-1/wal/instance
I20260926 07:30:15.078651 22743 external_mini_cluster.cc:1428] Setting key 17496752f530430e79fac4e393c07601
Sep 26 07:30:15 dist-test-slave-wdjc krb5kdc[28295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407815, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.194@KRBTEST.COM for kudu/127.22.53.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.22.53.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.53.195@KRBTEST.COM" created.
I20260926 07:30:15.092998 28379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:15.077137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.194:51661 (local address 127.22.53.254:41479)
0926 07:30:15.077292 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:15.077296 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:15.078391 (+ 1095us) server_negotiation.cc:407] Connection header received
0926 07:30:15.080195 (+ 1804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:15.080198 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:15.080257 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:15.080372 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:15.081887 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:15.082488 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:15.084482 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:15.084895 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:15.084999 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:15.088576 (+ 3577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:15.088585 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:15.088591 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:15.088633 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:15.090278 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:15.091012 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:15.091016 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:15.091017 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:15.091109 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:15.091520 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:15.091523 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:15.091524 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:15.091773 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:15.091878 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:15.092540 (+ 662us) server_negotiation.cc:299] Negotiation successful
0926 07:30:15.092755 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 07:30:15.093883 28667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:15.077508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:41479 (local address 127.22.53.194:51661)
0926 07:30:15.078158 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:15.078244 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:15.079918 (+ 1674us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:15.080653 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:15.080668 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:15.081083 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:15.081652 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:15.081675 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:15.082647 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:15.082654 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:15.084280 (+ 1626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:15.084297 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:15.085096 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:15.085101 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:15.085355 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:15.086173 (+ 818us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:15.086204 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:15.088197 (+ 1993us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:30:15.090428 (+ 2231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:15.090435 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:15.090452 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:15.090859 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:15.091217 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:15.091219 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:15.091220 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:15.091403 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:15.091906 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:30:15.091912 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:30:15.092214 (+ 302us) client_negotiation.cc:769] Sending connection context
0926 07:30:15.092402 (+ 188us) client_negotiation.cc:240] Negotiation successful
0926 07:30:15.092887 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":127,"threads_started":1}
I20260926 07:30:15.095283 28665 heartbeater.cc:347] Connected to a master server at 127.22.53.254:41479
I20260926 07:30:15.095597 28665 heartbeater.cc:464] Registering TS with master...
I20260926 07:30:15.096266 28665 heartbeater.cc:511] Master 127.22.53.254:41479 requested a full tablet report, sending...
I20260926 07:30:15.097585 28339 ts_manager.cc:194] Registered new tserver with Master: 1665934dc5c34c7ca4676464e02f915a (127.22.53.194:34703)
I20260926 07:30:15.098230 28339 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.53.194@KRBTEST.COM'} at 127.22.53.194:51661
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.53.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.195.keytab.
Entry for principal kudu/127.22.53.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.195.keytab.
I20260926 07:30:15.113162 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407783529717-22743-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407783529717-22743-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407783529717-22743-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407783529717-22743-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.53.195:0
--local_ip_for_outbound_sockets=127.22.53.195
--webserver_interface=127.22.53.195
--webserver_port=0
--tserver_master_addrs=127.22.53.254:41479
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.195.keytab
--principal=kudu/127.22.53.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:44813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:30:15.235854 28670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:30:15.236210 28670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:30:15.236383 28670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:30:15.241400 28670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:30:15.241508 28670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:30:15.241605 28670 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.53.195
I20260926 07:30:15.247462 28670 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:44813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407783529717-22743-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407783529717-22743-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.195.keytab
--principal=kudu/127.22.53.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.53.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407783529717-22743-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.53.195
--webserver_port=0
--tserver_master_addrs=127.22.53.254:41479
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28670
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.53.195
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407783529717-22743-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:30:15.248804 28670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:30:15.249900 28670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:30:15.260478 28676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:15.260480 28678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:15.260524 28670 server_base.cc:1034] running on GCE node
W20260926 07:30:15.260480 28675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:15.261478 28670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:30:15.262172 28670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:30:15.263365 28670 hybrid_clock.cc:648] HybridClock initialized: now 1790407815263333 us; error 45 us; skew 500 ppm
Sep 26 07:30:15 dist-test-slave-wdjc krb5kdc[28295](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407815, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:30:15.266593 28670 init.cc:377] Logged in from keytab as kudu/127.22.53.195@KRBTEST.COM (short username kudu)
I20260926 07:30:15.267634 28670 webserver.cc:460] Webserver started at http://127.22.53.195:42913/ using document root <none> and password file <none>
I20260926 07:30:15.268225 28670 fs_manager.cc:359] Metadata directory not provided
I20260926 07:30:15.268296 28670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:30:15.268477 28670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:30:15.270866 28670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407783529717-22743-0/minicluster-data/ts-2/data/instance:
uuid: "8f9124a6340243f9957261dabb900474"
format_stamp: "Formatted at 2026-09-26 07:30:15 on dist-test-slave-wdjc"
server_key: "302c3277122453426a5887999e48f2cc"
server_key_iv: "852e7822720e46ef2b51912a9a964d52"
server_key_version: "encryptionkey@0"
I20260926 07:30:15.271449 28670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407783529717-22743-0/minicluster-data/ts-2/wal/instance:
uuid: "8f9124a6340243f9957261dabb900474"
format_stamp: "Formatted at 2026-09-26 07:30:15 on dist-test-slave-wdjc"
server_key: "302c3277122453426a5887999e48f2cc"
server_key_iv: "852e7822720e46ef2b51912a9a964d52"
server_key_version: "encryptionkey@0"
I20260926 07:30:15.275722 28670 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 07:30:15.278786 28685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:15.280364 28670 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 07:30:15.280685 28670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407783529717-22743-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407783529717-22743-0/minicluster-data/ts-2/wal
uuid: "8f9124a6340243f9957261dabb900474"
format_stamp: "Formatted at 2026-09-26 07:30:15 on dist-test-slave-wdjc"
server_key: "302c3277122453426a5887999e48f2cc"
server_key_iv: "852e7822720e46ef2b51912a9a964d52"
server_key_version: "encryptionkey@0"
I20260926 07:30:15.280845 28670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407783529717-22743-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407783529717-22743-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407783529717-22743-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:30:15.309648 28670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:30:15.313580 28670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:30:15.313838 28670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:30:15.314527 28670 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:30:15.315503 28670 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:30:15.315555 28670 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:15.315630 28670 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:30:15.315680 28670 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:15.325704 28670 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.195:41157
I20260926 07:30:15.325735 28798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.195:41157 every 8 connection(s)
I20260926 07:30:15.326889 28670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407783529717-22743-0/minicluster-data/ts-2/data/info.pb
I20260926 07:30:15.331944 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 28670
I20260926 07:30:15.332108 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407783529717-22743-0/minicluster-data/ts-2/wal/instance
I20260926 07:30:15.332468 22743 external_mini_cluster.cc:1428] Setting key 1a06185d380e79684072adb3b462d8e6
Sep 26 07:30:15 dist-test-slave-wdjc krb5kdc[28295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407815, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.195@KRBTEST.COM for kudu/127.22.53.254@KRBTEST.COM
I20260926 07:30:15.343407 28379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:15.328964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.195:44199 (local address 127.22.53.254:41479)
0926 07:30:15.329158 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:15.329162 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:15.330150 (+ 988us) server_negotiation.cc:407] Connection header received
0926 07:30:15.331672 (+ 1522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:15.331678 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:15.331760 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:15.331881 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:15.333740 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:15.334510 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:15.336099 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:15.336346 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:15.336446 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:15.339410 (+ 2964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:15.339413 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:15.339415 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:15.339456 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:15.340926 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:15.341533 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:15.341536 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:15.341537 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:15.341720 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:15.342107 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:15.342110 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:15.342111 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:15.342383 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:15.342480 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:15.343140 (+ 660us) server_negotiation.cc:299] Negotiation successful
0926 07:30:15.343271 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260926 07:30:15.344310 28801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:15.329400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:41479 (local address 127.22.53.195:44199)
0926 07:30:15.329965 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:15.330016 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:15.331376 (+ 1360us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:15.332144 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:15.332163 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:15.332763 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:15.333483 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:15.333499 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:15.334671 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:15.334676 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:15.335948 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:15.335960 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:15.336520 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:15.336523 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:15.336786 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:15.337536 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:15.337554 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:15.339200 (+ 1646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:30:15.341049 (+ 1849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:15.341055 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:15.341071 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:15.341387 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:15.341832 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:15.341835 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:15.341837 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:15.341998 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:15.342497 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:30:15.342503 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:30:15.342823 (+ 320us) client_negotiation.cc:769] Sending connection context
0926 07:30:15.343009 (+ 186us) client_negotiation.cc:240] Negotiation successful
0926 07:30:15.343323 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":126,"threads_started":1}
I20260926 07:30:15.345727 28799 heartbeater.cc:347] Connected to a master server at 127.22.53.254:41479
I20260926 07:30:15.346024 28799 heartbeater.cc:464] Registering TS with master...
I20260926 07:30:15.346663 28799 heartbeater.cc:511] Master 127.22.53.254:41479 requested a full tablet report, sending...
I20260926 07:30:15.347761 28339 ts_manager.cc:194] Registered new tserver with Master: 8f9124a6340243f9957261dabb900474 (127.22.53.195:41157)
I20260926 07:30:15.348369 28339 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.53.195@KRBTEST.COM'} at 127.22.53.195:44199
I20260926 07:30:15.359500 22743 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407783529717-22743-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407783529717-22743-0/krb5kdc/test-admin.keytab.
Sep 26 07:30:15 dist-test-slave-wdjc krb5kdc[28295](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407815, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:30:15.377959 22743 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260926 07:30:15.381165 28806 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:30:15.379535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55488 (local address 127.22.53.193:39183)
0926 07:30:15.379910 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:15.379918 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:15.379949 (+ 31us) server_negotiation.cc:407] Connection header received
0926 07:30:15.380016 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:15.380023 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:15.380238 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:15.380411 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:15.381000 (+ 589us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55488: BlockingRecv error: recv got EOF from 127.0.0.1:55488 (error 108)
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":141,"threads_started":1}
W20260926 07:30:15.381446 22743 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407783529717-22743-0/krb5kdc/krb5cc' of type FILE
Sep 26 07:30:15 dist-test-slave-wdjc krb5kdc[28295](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407815, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:30:15.382666 22743 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 07:30:15 dist-test-slave-wdjc krb5kdc[28295](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407815, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:30:15.383859 22743 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 07:30:15 dist-test-slave-wdjc krb5kdc[28295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407815, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.53.193@KRBTEST.COM
I20260926 07:30:15.395550 28806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:15.385050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55490 (local address 127.22.53.193:39183)
0926 07:30:15.385194 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:15.385198 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:15.385312 (+ 114us) server_negotiation.cc:407] Connection header received
0926 07:30:15.385522 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:15.385525 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:15.385574 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:15.385646 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:15.386257 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:15.387032 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:15.388046 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:15.388648 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:15.388741 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:15.391363 (+ 2622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:15.391377 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:15.391393 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:15.391426 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:15.393372 (+ 1946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:15.394002 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:15.394007 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:15.394009 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:15.394105 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:15.394418 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:15.394421 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:15.394422 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:15.394738 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:15.394931 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:15.395158 (+ 227us) server_negotiation.cc:299] Negotiation successful
0926 07:30:15.395374 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260926 07:30:15.400264 28806 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:30:15.399023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55498 (local address 127.22.53.193:39183)
0926 07:30:15.399188 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:15.399193 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:15.399357 (+ 164us) server_negotiation.cc:407] Connection header received
0926 07:30:15.399471 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:15.399474 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:15.399517 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:15.399589 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:15.400151 (+ 562us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55498: BlockingRecv error: recv got EOF from 127.0.0.1:55498 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260926 07:30:15.400507 22743 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407783529717-22743-0/krb5kdc/krb5cc' of type FILE
Sep 26 07:30:15 dist-test-slave-wdjc krb5kdc[28295](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407815, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:30:15.401764 22743 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 07:30:15 dist-test-slave-wdjc krb5kdc[28295](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407815, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:30:15.402872 22743 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 07:30:15 dist-test-slave-wdjc krb5kdc[28295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407815, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.53.193@KRBTEST.COM
I20260926 07:30:15.413437 28806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:15.403998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55510 (local address 127.22.53.193:39183)
0926 07:30:15.404246 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:15.404253 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:15.404388 (+ 135us) server_negotiation.cc:407] Connection header received
0926 07:30:15.404613 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:15.404616 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:15.404664 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:15.404786 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:15.405359 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:15.405894 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:15.406777 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:15.407011 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:15.407128 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:15.409929 (+ 2801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:15.409935 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:15.409940 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:15.409988 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:15.411530 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:15.412048 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:15.412051 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:15.412052 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:15.412152 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:15.412405 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:15.412407 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:15.412408 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:15.412809 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:15.412943 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:15.413199 (+ 256us) server_negotiation.cc:299] Negotiation successful
0926 07:30:15.413310 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260926 07:30:15.416742 28806 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:30:15.415529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55522 (local address 127.22.53.193:39183)
0926 07:30:15.415682 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:15.415687 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:15.415797 (+ 110us) server_negotiation.cc:407] Connection header received
0926 07:30:15.415965 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:15.415970 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:15.416017 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:15.416119 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:15.416613 (+ 494us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55522: BlockingRecv error: recv got EOF from 127.0.0.1:55522 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260926 07:30:15.416993 22743 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407783529717-22743-0/krb5kdc/krb5cc' of type FILE
Sep 26 07:30:15 dist-test-slave-wdjc krb5kdc[28295](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407815, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:30:15.418170 22743 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 07:30:15 dist-test-slave-wdjc krb5kdc[28295](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407815, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:30:15.419144 22743 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 07:30:15 dist-test-slave-wdjc krb5kdc[28295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407815, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.53.193@KRBTEST.COM
I20260926 07:30:15.430122 28806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:15.420192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55526 (local address 127.22.53.193:39183)
0926 07:30:15.420374 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:15.420378 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:15.420498 (+ 120us) server_negotiation.cc:407] Connection header received
0926 07:30:15.420743 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:15.420747 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:15.420793 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:15.420892 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:15.421485 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:15.422044 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:15.422929 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:15.423163 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:15.423289 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:15.426292 (+ 3003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:15.426297 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:15.426299 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:15.426335 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:15.427988 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:15.428716 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:15.428719 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:15.428720 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:15.428819 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:15.429153 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:15.429155 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:15.429156 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:15.429354 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:15.429492 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:15.429760 (+ 268us) server_negotiation.cc:299] Negotiation successful
0926 07:30:15.429913 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 07:30:15.431298 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 28402
I20260926 07:30:15.438800 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 28536
I20260926 07:30:15.447258 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 28670
I20260926 07:30:15.454357 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 28308
2026-09-26T07:30:15Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2398 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:30:15 dist-test-slave-wdjc krb5kdc[28819](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:30:15 dist-test-slave-wdjc krb5kdc[28819](info): set up 2 sockets
Sep 26 07:30:15 dist-test-slave-wdjc krb5kdc[28819](info): commencing operation
krb5kdc: starting...
W20260926 07:30:16.578053 22743 mini_kdc.cc:121] Time spent starting KDC: real 1.100s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/krb5kdc/test-user.keytab.
Sep 26 07:30:16 dist-test-slave-wdjc krb5kdc[28819](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407816, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:30:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:30:16Z Disabled control of system clock
WARNING: no policy specified for oryx/127.22.53.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.53.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.53.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/krb5kdc/oryx_127.22.53.254.keytab.
Entry for principal oryx/127.22.53.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/krb5kdc/oryx_127.22.53.254.keytab.
I20260926 07:30:16.723907 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.53.254:44679
--webserver_interface=127.22.53.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/krb5kdc/oryx_127.22.53.254.keytab
--principal=oryx/127.22.53.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:40785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.53.254:44679
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:30:16.842454 28832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:30:16.842836 28832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:30:16.842919 28832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:30:16.847334 28832 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:30:16.847456 28832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:30:16.847496 28832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:30:16.847535 28832 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:30:16.847568 28832 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:30:16.853633 28832 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:40785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.53.254:44679
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/krb5kdc/oryx_127.22.53.254.keytab
--principal=oryx/127.22.53.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.53.254:44679
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.53.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28832
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:30:16.854854 28832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:30:16.855927 28832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:30:16.864928 28840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:16.864954 28838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:16.865236 28832 server_base.cc:1034] running on GCE node
W20260926 07:30:16.864938 28837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:16.865932 28832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:30:16.867164 28832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:30:16.868383 28832 hybrid_clock.cc:648] HybridClock initialized: now 1790407816868356 us; error 41 us; skew 500 ppm
Sep 26 07:30:16 dist-test-slave-wdjc krb5kdc[28819](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407816, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.53.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:30:16.875511 28832 init.cc:377] Logged in from keytab as oryx/127.22.53.254@KRBTEST.COM (short username oryx)
I20260926 07:30:16.876744 28832 webserver.cc:460] Webserver started at http://127.22.53.254:33597/ using document root <none> and password file <none>
I20260926 07:30:16.877373 28832 fs_manager.cc:359] Metadata directory not provided
I20260926 07:30:16.877475 28832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:30:16.877753 28832 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:30:16.880003 28832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/master-0/data/instance:
uuid: "b71a0534077f44f7b2cd8a7fa7688789"
format_stamp: "Formatted at 2026-09-26 07:30:16 on dist-test-slave-wdjc"
server_key: "8d84e2f5fd281795ab6bf9fd9488fa1c"
server_key_iv: "446571a55616c1034d927f8a70229c77"
server_key_version: "encryptionkey@0"
I20260926 07:30:16.880738 28832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/master-0/wal/instance:
uuid: "b71a0534077f44f7b2cd8a7fa7688789"
format_stamp: "Formatted at 2026-09-26 07:30:16 on dist-test-slave-wdjc"
server_key: "8d84e2f5fd281795ab6bf9fd9488fa1c"
server_key_iv: "446571a55616c1034d927f8a70229c77"
server_key_version: "encryptionkey@0"
I20260926 07:30:16.884934 28832 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 07:30:16.887753 28847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:16.889281 28832 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:30:16.889451 28832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/master-0/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/master-0/wal
uuid: "b71a0534077f44f7b2cd8a7fa7688789"
format_stamp: "Formatted at 2026-09-26 07:30:16 on dist-test-slave-wdjc"
server_key: "8d84e2f5fd281795ab6bf9fd9488fa1c"
server_key_iv: "446571a55616c1034d927f8a70229c77"
server_key_version: "encryptionkey@0"
I20260926 07:30:16.889596 28832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:30:16.932195 28832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:30:16.935531 28832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:30:16.935858 28832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:30:16.943521 28832 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.254:44679
I20260926 07:30:16.943866 28899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.254:44679 every 8 connection(s)
I20260926 07:30:16.944710 28832 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
I20260926 07:30:16.947608 28900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:30:16.951930 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 28832
I20260926 07:30:16.952117 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/master-0/wal/instance
I20260926 07:30:16.952430 22743 external_mini_cluster.cc:1428] Setting key a7aec8dfd7023dbf8141d3d7bea2d036
I20260926 07:30:16.953992 28900 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b71a0534077f44f7b2cd8a7fa7688789: Bootstrap starting.
I20260926 07:30:16.956984 28900 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b71a0534077f44f7b2cd8a7fa7688789: Neither blocks nor log segments found. Creating new log.
I20260926 07:30:16.958106 28900 log.cc:824] T 00000000000000000000000000000000 P b71a0534077f44f7b2cd8a7fa7688789: Log is configured to *not* fsync() on all Append() calls
I20260926 07:30:16.962076 28900 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b71a0534077f44f7b2cd8a7fa7688789: No bootstrap required, opened a new log
Sep 26 07:30:16 dist-test-slave-wdjc krb5kdc[28819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407816, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM
I20260926 07:30:16.967960 28900 raft_consensus.cc:348] T 00000000000000000000000000000000 P b71a0534077f44f7b2cd8a7fa7688789 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b71a0534077f44f7b2cd8a7fa7688789" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 44679 } }
I20260926 07:30:16.968372 28900 raft_consensus.cc:374] T 00000000000000000000000000000000 P b71a0534077f44f7b2cd8a7fa7688789 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:30:16.968428 28900 raft_consensus.cc:729] T 00000000000000000000000000000000 P b71a0534077f44f7b2cd8a7fa7688789 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b71a0534077f44f7b2cd8a7fa7688789, State: Initialized, Role: FOLLOWER
I20260926 07:30:16.969010 28900 consensus_queue.cc:260] T 00000000000000000000000000000000 P b71a0534077f44f7b2cd8a7fa7688789 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b71a0534077f44f7b2cd8a7fa7688789" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 44679 } }
I20260926 07:30:16.969223 28900 raft_consensus.cc:388] T 00000000000000000000000000000000 P b71a0534077f44f7b2cd8a7fa7688789 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:30:16.969574 28900 raft_consensus.cc:482] T 00000000000000000000000000000000 P b71a0534077f44f7b2cd8a7fa7688789 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:30:16.969696 28900 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b71a0534077f44f7b2cd8a7fa7688789 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:30:16.971710 28900 raft_consensus.cc:504] T 00000000000000000000000000000000 P b71a0534077f44f7b2cd8a7fa7688789 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b71a0534077f44f7b2cd8a7fa7688789" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 44679 } }
I20260926 07:30:16.972239 28900 leader_election.cc:302] T 00000000000000000000000000000000 P b71a0534077f44f7b2cd8a7fa7688789 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b71a0534077f44f7b2cd8a7fa7688789; no voters:
I20260926 07:30:16.972680 28900 leader_election.cc:288] T 00000000000000000000000000000000 P b71a0534077f44f7b2cd8a7fa7688789 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:30:16.973152 28903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:16.954191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44702 (local address 127.22.53.254:44679)
0926 07:30:16.954664 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:16.954683 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:16.954738 (+ 55us) server_negotiation.cc:407] Connection header received
0926 07:30:16.955655 (+ 917us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:16.955681 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:16.956130 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:16.956526 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:16.957532 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:16.958932 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:16.959861 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:16.960759 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:16.960955 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:16.963854 (+ 2899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:16.963862 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:16.963883 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:16.963923 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:16.968377 (+ 4454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:16.969169 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:16.969178 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:16.969184 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:16.969676 (+ 492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:16.970024 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:16.970028 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:16.970029 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:16.970646 (+ 617us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:16.971145 (+ 499us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:16.971520 (+ 375us) server_negotiation.cc:299] Negotiation successful
0926 07:30:16.972009 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":140,"threads_started":1}
I20260926 07:30:16.973167 28906 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b71a0534077f44f7b2cd8a7fa7688789 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:30:16.974087 28900 sys_catalog.cc:564] T 00000000000000000000000000000000 P b71a0534077f44f7b2cd8a7fa7688789 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:30:16.975005 28906 raft_consensus.cc:686] T 00000000000000000000000000000000 P b71a0534077f44f7b2cd8a7fa7688789 [term 1 LEADER]: Becoming Leader. State: Replica: b71a0534077f44f7b2cd8a7fa7688789, State: Running, Role: LEADER
I20260926 07:30:16.975412 28906 consensus_queue.cc:237] T 00000000000000000000000000000000 P b71a0534077f44f7b2cd8a7fa7688789 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b71a0534077f44f7b2cd8a7fa7688789" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 44679 } }
I20260926 07:30:16.976883 28910 sys_catalog.cc:455] T 00000000000000000000000000000000 P b71a0534077f44f7b2cd8a7fa7688789 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b71a0534077f44f7b2cd8a7fa7688789. Latest consensus state: current_term: 1 leader_uuid: "b71a0534077f44f7b2cd8a7fa7688789" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b71a0534077f44f7b2cd8a7fa7688789" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 44679 } } }
I20260926 07:30:16.977270 28905 sys_catalog.cc:455] T 00000000000000000000000000000000 P b71a0534077f44f7b2cd8a7fa7688789 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b71a0534077f44f7b2cd8a7fa7688789" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b71a0534077f44f7b2cd8a7fa7688789" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 44679 } } }
I20260926 07:30:16.977046 28910 sys_catalog.cc:458] T 00000000000000000000000000000000 P b71a0534077f44f7b2cd8a7fa7688789 [sys.catalog]: This master's current role is: LEADER
I20260926 07:30:16.977389 28905 sys_catalog.cc:458] T 00000000000000000000000000000000 P b71a0534077f44f7b2cd8a7fa7688789 [sys.catalog]: This master's current role is: LEADER
W20260926 07:30:16.980175 28919 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b71a0534077f44f7b2cd8a7fa7688789: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:30:16.980283 28919 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:30:16.980376 28920 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:30:16.981279 28920 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:30:16.987910 28920 catalog_manager.cc:1347] Generated new cluster ID: 8a78e9ea9d25425092fd72f238f444f5
I20260926 07:30:16.987999 28920 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:30:17.014183 28920 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:30:17.015620 28920 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:30:17.026657 28920 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b71a0534077f44f7b2cd8a7fa7688789: Generated new TSK 0
I20260926 07:30:17.027539 28920 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.22.53.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.53.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.53.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/krb5kdc/oryx_127.22.53.193.keytab.
Entry for principal oryx/127.22.53.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/krb5kdc/oryx_127.22.53.193.keytab.
I20260926 07:30:17.073210 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.53.193:0
--local_ip_for_outbound_sockets=127.22.53.193
--webserver_interface=127.22.53.193
--webserver_port=0
--tserver_master_addrs=127.22.53.254:44679
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/krb5kdc/oryx_127.22.53.193.keytab
--principal=oryx/127.22.53.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:40785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:30:17.191591 28926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:30:17.191898 28926 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:30:17.191946 28926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:30:17.196646 28926 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:30:17.196729 28926 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:30:17.196794 28926 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.53.193
I20260926 07:30:17.202445 28926 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:40785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/krb5kdc/oryx_127.22.53.193.keytab
--principal=oryx/127.22.53.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.53.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.53.193
--webserver_port=0
--tserver_master_addrs=127.22.53.254:44679
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28926
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.53.193
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:30:17.203541 28926 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:30:17.204592 28926 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:30:17.214726 28932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:17.214726 28934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:17.214759 28931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:17.215081 28926 server_base.cc:1034] running on GCE node
I20260926 07:30:17.215577 28926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:30:17.216338 28926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:30:17.217515 28926 hybrid_clock.cc:648] HybridClock initialized: now 1790407817217501 us; error 43 us; skew 500 ppm
Sep 26 07:30:17 dist-test-slave-wdjc krb5kdc[28819](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407817, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.53.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:30:17.220397 28926 init.cc:377] Logged in from keytab as oryx/127.22.53.193@KRBTEST.COM (short username oryx)
I20260926 07:30:17.221450 28926 webserver.cc:460] Webserver started at http://127.22.53.193:40931/ using document root <none> and password file <none>
I20260926 07:30:17.222055 28926 fs_manager.cc:359] Metadata directory not provided
I20260926 07:30:17.222119 28926 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:30:17.222321 28926 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:30:17.224720 28926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/ts-0/data/instance:
uuid: "96b38667cb7946819bcb4dfc480f93be"
format_stamp: "Formatted at 2026-09-26 07:30:17 on dist-test-slave-wdjc"
server_key: "93355256a82da45aebaecc8b8e52aac1"
server_key_iv: "b8dcbaae9d8be84df493735cc9823431"
server_key_version: "encryptionkey@0"
I20260926 07:30:17.225310 28926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/ts-0/wal/instance:
uuid: "96b38667cb7946819bcb4dfc480f93be"
format_stamp: "Formatted at 2026-09-26 07:30:17 on dist-test-slave-wdjc"
server_key: "93355256a82da45aebaecc8b8e52aac1"
server_key_iv: "b8dcbaae9d8be84df493735cc9823431"
server_key_version: "encryptionkey@0"
I20260926 07:30:17.229506 28926 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 07:30:17.232314 28941 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:17.233791 28926 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 07:30:17.233915 28926 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/ts-0/wal
uuid: "96b38667cb7946819bcb4dfc480f93be"
format_stamp: "Formatted at 2026-09-26 07:30:17 on dist-test-slave-wdjc"
server_key: "93355256a82da45aebaecc8b8e52aac1"
server_key_iv: "b8dcbaae9d8be84df493735cc9823431"
server_key_version: "encryptionkey@0"
I20260926 07:30:17.234023 28926 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:30:17.275915 28926 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:30:17.279404 28926 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:30:17.279580 28926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:30:17.280148 28926 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:30:17.281109 28926 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:30:17.281159 28926 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:17.281201 28926 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:30:17.281226 28926 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:17.291206 28926 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.193:35569
I20260926 07:30:17.291337 29054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.193:35569 every 8 connection(s)
I20260926 07:30:17.292248 28926 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/ts-0/data/info.pb
I20260926 07:30:17.301666 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 28926
I20260926 07:30:17.301833 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/ts-0/wal/instance
I20260926 07:30:17.302168 22743 external_mini_cluster.cc:1428] Setting key b91f787c82078e70c184e6a1a47880eb
Sep 26 07:30:17 dist-test-slave-wdjc krb5kdc[28819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407817, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.53.193@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM
I20260926 07:30:17.308154 28903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:17.294345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.193:40145 (local address 127.22.53.254:44679)
0926 07:30:17.294498 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:17.294502 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:17.295472 (+ 970us) server_negotiation.cc:407] Connection header received
0926 07:30:17.296939 (+ 1467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:17.296943 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:17.297008 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:17.297089 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:17.298369 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:17.298928 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:17.300152 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:17.300398 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:17.300521 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:17.303534 (+ 3013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:17.303538 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:17.303540 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:17.303565 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:17.305498 (+ 1933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:17.306132 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:17.306138 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:17.306142 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:17.306261 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:17.306665 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:17.306671 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:17.306675 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:17.306943 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:17.307054 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:17.307755 (+ 701us) server_negotiation.cc:299] Negotiation successful
0926 07:30:17.307927 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 07:30:17.309330 29057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:17.294789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:44679 (local address 127.22.53.193:40145)
0926 07:30:17.295293 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:17.295340 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:17.296702 (+ 1362us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:17.297282 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:17.297290 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:17.297678 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:17.298179 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:17.298193 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:17.299080 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:17.299088 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:17.300006 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:17.300016 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:17.300610 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:17.300615 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:17.300794 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:17.301500 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:17.301523 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:17.303341 (+ 1818us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:30:17.305674 (+ 2333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:17.305680 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:17.305697 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:17.306001 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:17.306386 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:17.306389 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:17.306390 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:17.306557 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:17.307096 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:30:17.307103 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:30:17.307398 (+ 295us) client_negotiation.cc:769] Sending connection context
0926 07:30:17.307687 (+ 289us) client_negotiation.cc:240] Negotiation successful
0926 07:30:17.308113 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":138,"threads_started":1}
I20260926 07:30:17.310890 29055 heartbeater.cc:347] Connected to a master server at 127.22.53.254:44679
I20260926 07:30:17.311183 29055 heartbeater.cc:464] Registering TS with master...
I20260926 07:30:17.311880 29055 heartbeater.cc:511] Master 127.22.53.254:44679 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.22.53.194@KRBTEST.COM; defaulting to no policy
I20260926 07:30:17.313755 28864 ts_manager.cc:194] Registered new tserver with Master: 96b38667cb7946819bcb4dfc480f93be (127.22.53.193:35569)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.53.194@KRBTEST.COM" created.
I20260926 07:30:17.315059 28864 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.53.193@KRBTEST.COM'} at 127.22.53.193:40145
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.53.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/krb5kdc/oryx_127.22.53.194.keytab.
Entry for principal oryx/127.22.53.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/krb5kdc/oryx_127.22.53.194.keytab.
I20260926 07:30:17.331476 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.53.194:0
--local_ip_for_outbound_sockets=127.22.53.194
--webserver_interface=127.22.53.194
--webserver_port=0
--tserver_master_addrs=127.22.53.254:44679
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/krb5kdc/oryx_127.22.53.194.keytab
--principal=oryx/127.22.53.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:40785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:30:17.447153 29060 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:30:17.447443 29060 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:30:17.447489 29060 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:30:17.451720 29060 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:30:17.451849 29060 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:30:17.451967 29060 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.53.194
I20260926 07:30:17.458868 29060 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:40785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/krb5kdc/oryx_127.22.53.194.keytab
--principal=oryx/127.22.53.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.53.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.53.194
--webserver_port=0
--tserver_master_addrs=127.22.53.254:44679
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29060
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.53.194
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:30:17.460067 29060 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:30:17.461215 29060 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 07:30:17.471747 29060 server_base.cc:1034] running on GCE node
W20260926 07:30:17.471668 29066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:17.471668 29068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:17.471668 29065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:17.472728 29060 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:30:17.473502 29060 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:30:17.474700 29060 hybrid_clock.cc:648] HybridClock initialized: now 1790407817474682 us; error 30 us; skew 500 ppm
Sep 26 07:30:17 dist-test-slave-wdjc krb5kdc[28819](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407817, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.53.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:30:17.478085 29060 init.cc:377] Logged in from keytab as oryx/127.22.53.194@KRBTEST.COM (short username oryx)
I20260926 07:30:17.479095 29060 webserver.cc:460] Webserver started at http://127.22.53.194:37591/ using document root <none> and password file <none>
I20260926 07:30:17.479666 29060 fs_manager.cc:359] Metadata directory not provided
I20260926 07:30:17.479734 29060 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:30:17.479914 29060 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:30:17.482112 29060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/ts-1/data/instance:
uuid: "ddbc2223fead4ddfa34d0a137efe01cb"
format_stamp: "Formatted at 2026-09-26 07:30:17 on dist-test-slave-wdjc"
server_key: "67a6896d69677c2372a738d44458e3b3"
server_key_iv: "77be86c71443834c1b7982e75bca4723"
server_key_version: "encryptionkey@0"
I20260926 07:30:17.482832 29060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/ts-1/wal/instance:
uuid: "ddbc2223fead4ddfa34d0a137efe01cb"
format_stamp: "Formatted at 2026-09-26 07:30:17 on dist-test-slave-wdjc"
server_key: "67a6896d69677c2372a738d44458e3b3"
server_key_iv: "77be86c71443834c1b7982e75bca4723"
server_key_version: "encryptionkey@0"
I20260926 07:30:17.486928 29060 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 07:30:17.489635 29075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:17.491004 29060 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:30:17.491164 29060 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/ts-1/wal
uuid: "ddbc2223fead4ddfa34d0a137efe01cb"
format_stamp: "Formatted at 2026-09-26 07:30:17 on dist-test-slave-wdjc"
server_key: "67a6896d69677c2372a738d44458e3b3"
server_key_iv: "77be86c71443834c1b7982e75bca4723"
server_key_version: "encryptionkey@0"
I20260926 07:30:17.491297 29060 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:30:17.505234 29060 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:30:17.508683 29060 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:30:17.508900 29060 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:30:17.509529 29060 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:30:17.510433 29060 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:30:17.510481 29060 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:17.510522 29060 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:30:17.510538 29060 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:17.520853 29060 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.194:38993
I20260926 07:30:17.520968 29188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.194:38993 every 8 connection(s)
I20260926 07:30:17.521921 29060 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/ts-1/data/info.pb
I20260926 07:30:17.528520 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 29060
I20260926 07:30:17.528669 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/ts-1/wal/instance
I20260926 07:30:17.528966 22743 external_mini_cluster.cc:1428] Setting key 4d8ca347434d5609588d12fe6e72c999
Sep 26 07:30:17 dist-test-slave-wdjc krb5kdc[28819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407817, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.53.194@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM
I20260926 07:30:17.538141 28903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:17.523955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.194:49573 (local address 127.22.53.254:44679)
0926 07:30:17.524112 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:17.524116 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:17.525064 (+ 948us) server_negotiation.cc:407] Connection header received
0926 07:30:17.526555 (+ 1491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:17.526558 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:17.526623 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:17.526733 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:17.528042 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:17.528811 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:17.530255 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:17.530577 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:17.530692 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:17.533628 (+ 2936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:17.533634 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:17.533641 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:17.533703 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:17.535428 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:17.536068 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:17.536073 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:17.536078 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:17.536191 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:17.536628 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:17.536633 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:17.536637 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:17.536979 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:17.537094 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:17.537735 (+ 641us) server_negotiation.cc:299] Negotiation successful
0926 07:30:17.537911 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 07:30:17.538995 29191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:17.524360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:44679 (local address 127.22.53.194:49573)
0926 07:30:17.524887 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:17.524933 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:17.526309 (+ 1376us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:17.526953 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:17.526961 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:17.527378 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:17.527883 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:17.527897 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:17.528957 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:17.528960 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:17.530117 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:17.530129 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:17.530746 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:17.530749 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:17.530912 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:17.531650 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:17.531668 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:17.533434 (+ 1766us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:30:17.535594 (+ 2160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:17.535600 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:17.535617 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:17.535943 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:17.536308 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:17.536311 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:17.536312 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:17.536472 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:17.537094 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:30:17.537100 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:30:17.537385 (+ 285us) client_negotiation.cc:769] Sending connection context
0926 07:30:17.537604 (+ 219us) client_negotiation.cc:240] Negotiation successful
0926 07:30:17.537950 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":147,"threads_started":1}
WARNING: no policy specified for oryx/127.22.53.195@KRBTEST.COM; defaulting to no policy
I20260926 07:30:17.540362 29189 heartbeater.cc:347] Connected to a master server at 127.22.53.254:44679
I20260926 07:30:17.540688 29189 heartbeater.cc:464] Registering TS with master...
I20260926 07:30:17.541271 29189 heartbeater.cc:511] Master 127.22.53.254:44679 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.53.195@KRBTEST.COM" created.
I20260926 07:30:17.542675 28864 ts_manager.cc:194] Registered new tserver with Master: ddbc2223fead4ddfa34d0a137efe01cb (127.22.53.194:38993)
I20260926 07:30:17.543542 28864 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.53.194@KRBTEST.COM'} at 127.22.53.194:49573
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.53.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/krb5kdc/oryx_127.22.53.195.keytab.
Entry for principal oryx/127.22.53.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/krb5kdc/oryx_127.22.53.195.keytab.
I20260926 07:30:17.559887 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.53.195:0
--local_ip_for_outbound_sockets=127.22.53.195
--webserver_interface=127.22.53.195
--webserver_port=0
--tserver_master_addrs=127.22.53.254:44679
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/krb5kdc/oryx_127.22.53.195.keytab
--principal=oryx/127.22.53.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:40785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:30:17.676944 29194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:30:17.677253 29194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:30:17.677299 29194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:30:17.681807 29194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:30:17.681891 29194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:30:17.681955 29194 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.53.195
I20260926 07:30:17.687834 29194 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:40785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/krb5kdc/oryx_127.22.53.195.keytab
--principal=oryx/127.22.53.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.53.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.53.195
--webserver_port=0
--tserver_master_addrs=127.22.53.254:44679
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29194
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.53.195
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:30:17.689208 29194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:30:17.690331 29194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:30:17.700871 29202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:17.700871 29199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:17.700971 29200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:17.701004 29194 server_base.cc:1034] running on GCE node
I20260926 07:30:17.701629 29194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:30:17.702472 29194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:30:17.703641 29194 hybrid_clock.cc:648] HybridClock initialized: now 1790407817703627 us; error 28 us; skew 500 ppm
Sep 26 07:30:17 dist-test-slave-wdjc krb5kdc[28819](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407817, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.53.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:30:17.706635 29194 init.cc:377] Logged in from keytab as oryx/127.22.53.195@KRBTEST.COM (short username oryx)
I20260926 07:30:17.707572 29194 webserver.cc:460] Webserver started at http://127.22.53.195:33573/ using document root <none> and password file <none>
I20260926 07:30:17.708158 29194 fs_manager.cc:359] Metadata directory not provided
I20260926 07:30:17.708225 29194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:30:17.708405 29194 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:30:17.710686 29194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/ts-2/data/instance:
uuid: "ebec0858532b452191c80c0652eb9b8e"
format_stamp: "Formatted at 2026-09-26 07:30:17 on dist-test-slave-wdjc"
server_key: "7d3929fcb2ef50584f4326f146df8df2"
server_key_iv: "0463dc008ae7bc0f710c0734c10df298"
server_key_version: "encryptionkey@0"
I20260926 07:30:17.711266 29194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/ts-2/wal/instance:
uuid: "ebec0858532b452191c80c0652eb9b8e"
format_stamp: "Formatted at 2026-09-26 07:30:17 on dist-test-slave-wdjc"
server_key: "7d3929fcb2ef50584f4326f146df8df2"
server_key_iv: "0463dc008ae7bc0f710c0734c10df298"
server_key_version: "encryptionkey@0"
I20260926 07:30:17.715423 29194 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 07:30:17.718334 29209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:17.719815 29194 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 07:30:17.719983 29194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/ts-2/wal
uuid: "ebec0858532b452191c80c0652eb9b8e"
format_stamp: "Formatted at 2026-09-26 07:30:17 on dist-test-slave-wdjc"
server_key: "7d3929fcb2ef50584f4326f146df8df2"
server_key_iv: "0463dc008ae7bc0f710c0734c10df298"
server_key_version: "encryptionkey@0"
I20260926 07:30:17.720129 29194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:30:17.734314 29194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:30:17.738092 29194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:30:17.738314 29194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:30:17.738992 29194 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:30:17.739889 29194 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:30:17.739938 29194 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:17.740008 29194 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:30:17.740048 29194 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:17.749475 29194 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.195:42015
I20260926 07:30:17.749492 29322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.195:42015 every 8 connection(s)
I20260926 07:30:17.750491 29194 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/ts-2/data/info.pb
I20260926 07:30:17.757407 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 29194
I20260926 07:30:17.757547 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/ts-2/wal/instance
I20260926 07:30:17.757858 22743 external_mini_cluster.cc:1428] Setting key 571303d698c57a7265690cdb6cf5a7d8
Sep 26 07:30:17 dist-test-slave-wdjc krb5kdc[28819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407817, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.53.195@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM
I20260926 07:30:17.766594 28903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:17.752480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.195:59627 (local address 127.22.53.254:44679)
0926 07:30:17.752685 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:17.752690 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:17.753716 (+ 1026us) server_negotiation.cc:407] Connection header received
0926 07:30:17.755246 (+ 1530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:17.755249 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:17.755309 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:17.755396 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:17.756916 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:17.757804 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:17.759347 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:17.759572 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:17.759702 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:17.762559 (+ 2857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:17.762563 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:17.762565 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:17.762593 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:17.764201 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:17.764928 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:17.764931 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:17.764932 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:17.765014 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:17.765405 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:17.765408 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:17.765410 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:17.765621 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:17.765735 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:17.766281 (+ 546us) server_negotiation.cc:299] Negotiation successful
0926 07:30:17.766423 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260926 07:30:17.767479 29325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:17.752954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:44679 (local address 127.22.53.195:59627)
0926 07:30:17.753527 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:17.753577 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:17.754880 (+ 1303us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:17.755597 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:17.755606 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:17.756044 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:17.756713 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:17.756738 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:17.757988 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:17.757991 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:17.759192 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:17.759204 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:17.759762 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:17.759767 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:17.759962 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:17.760732 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:17.760750 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:17.762330 (+ 1580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:30:17.764376 (+ 2046us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:17.764383 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:17.764402 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:17.764802 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:17.765114 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:17.765117 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:17.765119 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:17.765296 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:17.765725 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:30:17.765733 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:30:17.766013 (+ 280us) client_negotiation.cc:769] Sending connection context
0926 07:30:17.766232 (+ 219us) client_negotiation.cc:240] Negotiation successful
0926 07:30:17.766591 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":146,"threads_started":1}
I20260926 07:30:17.769012 29323 heartbeater.cc:347] Connected to a master server at 127.22.53.254:44679
I20260926 07:30:17.769312 29323 heartbeater.cc:464] Registering TS with master...
I20260926 07:30:17.769924 29323 heartbeater.cc:511] Master 127.22.53.254:44679 requested a full tablet report, sending...
I20260926 07:30:17.771323 28864 ts_manager.cc:194] Registered new tserver with Master: ebec0858532b452191c80c0652eb9b8e (127.22.53.195:42015)
I20260926 07:30:17.772153 28864 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.53.195@KRBTEST.COM'} at 127.22.53.195:59627
I20260926 07:30:17.772418 22743 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.22.53.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.53.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.53.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/krb5kdc/oryx_127.22.53.253.keytab.
Entry for principal oryx/127.22.53.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/krb5kdc/oryx_127.22.53.253.keytab.
I20260926 07:30:17.799878 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.53.253:45889
--webserver_interface=127.22.53.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/krb5kdc/oryx_127.22.53.253.keytab
--principal=oryx/127.22.53.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:40785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.53.254:44679,127.22.53.253:45889
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:30:17.917514 29328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:30:17.917860 29328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:30:17.917945 29328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:30:17.922580 29328 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:30:17.922765 29328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:30:17.922811 29328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:30:17.922851 29328 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:30:17.922886 29328 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:30:17.928823 29328 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:40785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.22.53.254:44679,127.22.53.253:45889
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/krb5kdc/oryx_127.22.53.253.keytab
--principal=oryx/127.22.53.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.53.253:45889
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.22.53.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29328
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
W20260926 07:30:17.929148 29328 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.22.53.254:44679,127.22.53.253:45889'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260926 07:30:17.930147 29328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:30:17.931226 29328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:30:17.940632 29336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:17.940630 29334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:17.940693 29328 server_base.cc:1034] running on GCE node
W20260926 07:30:17.940639 29333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:17.941457 29328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:30:17.942708 29328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:30:17.943912 29328 hybrid_clock.cc:648] HybridClock initialized: now 1790407817943899 us; error 28 us; skew 500 ppm
Sep 26 07:30:17 dist-test-slave-wdjc krb5kdc[28819](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407817, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.53.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:30:17.947021 29328 init.cc:377] Logged in from keytab as oryx/127.22.53.253@KRBTEST.COM (short username oryx)
I20260926 07:30:17.948156 29328 webserver.cc:460] Webserver started at http://127.22.53.253:45497/ using document root <none> and password file <none>
I20260926 07:30:17.948866 29328 fs_manager.cc:359] Metadata directory not provided
I20260926 07:30:17.948968 29328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:30:17.949218 29328 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:30:17.951452 29328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/master-1/data/instance:
uuid: "5343811b11a34d0483a88e2adef7fc12"
format_stamp: "Formatted at 2026-09-26 07:30:17 on dist-test-slave-wdjc"
server_key: "604acb043a909af244ce155ab1e927d3"
server_key_iv: "6e8511f4e42c03e9983acc3bcb248031"
server_key_version: "encryptionkey@0"
I20260926 07:30:17.952123 29328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/master-1/wal/instance:
uuid: "5343811b11a34d0483a88e2adef7fc12"
format_stamp: "Formatted at 2026-09-26 07:30:17 on dist-test-slave-wdjc"
server_key: "604acb043a909af244ce155ab1e927d3"
server_key_iv: "6e8511f4e42c03e9983acc3bcb248031"
server_key_version: "encryptionkey@0"
I20260926 07:30:17.956382 29328 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260926 07:30:17.959388 29343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:17.961030 29328 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:30:17.961203 29328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/master-1/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/master-1/wal
uuid: "5343811b11a34d0483a88e2adef7fc12"
format_stamp: "Formatted at 2026-09-26 07:30:17 on dist-test-slave-wdjc"
server_key: "604acb043a909af244ce155ab1e927d3"
server_key_iv: "6e8511f4e42c03e9983acc3bcb248031"
server_key_version: "encryptionkey@0"
I20260926 07:30:17.961342 29328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:30:17.976708 29328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:30:17.979861 29328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:30:17.980086 29328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:30:17.987284 29328 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.253:45889
I20260926 07:30:17.987663 29395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.253:45889 every 8 connection(s)
I20260926 07:30:17.988373 29328 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/master-1/data/info.pb
I20260926 07:30:17.991319 29396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:30:17.995421 29396 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.53.254" port: 44679 } has no permanent_uuid. Determining permanent_uuid...
I20260926 07:30:17.997287 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 29328
Sep 26 07:30:18 dist-test-slave-wdjc krb5kdc[28819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407817, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.53.253@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM
Sep 26 07:30:18 dist-test-slave-wdjc krb5kdc[28819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407816, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.22.53.253@KRBTEST.COM
I20260926 07:30:18.010095 28903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:17.997518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44706 (local address 127.22.53.254:44679)
0926 07:30:17.997664 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:17.997668 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:17.998858 (+ 1190us) server_negotiation.cc:407] Connection header received
0926 07:30:17.999694 (+ 836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:17.999696 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:17.999746 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:17.999816 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:18.000836 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:18.001451 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:18.002605 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:18.002823 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:18.002900 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:18.005707 (+ 2807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:18.005714 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:18.005720 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:18.005760 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:18.007568 (+ 1808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:18.008165 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:18.008170 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:18.008175 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:18.008277 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:18.008750 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:18.008753 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:18.008755 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:18.009072 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:18.009171 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:18.009763 (+ 592us) server_negotiation.cc:299] Negotiation successful
0926 07:30:18.009909 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 07:30:18.010980 29400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:17.998176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34524 (local address 127.22.53.253:45889)
0926 07:30:17.998532 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:17.998551 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:17.998622 (+ 71us) server_negotiation.cc:407] Connection header received
0926 07:30:17.999478 (+ 856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:17.999503 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:17.999746 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:17.999941 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:18.000673 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:18.001673 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:18.002475 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:18.002931 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:18.003094 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:18.006166 (+ 3072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:18.006172 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:18.006184 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:18.006239 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:18.008143 (+ 1904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:18.008676 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:18.008718 (+ 42us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:18.008722 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:18.008823 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:18.009098 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:18.009104 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:18.009109 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:18.009443 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:18.009600 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:18.009884 (+ 284us) server_negotiation.cc:299] Negotiation successful
0926 07:30:18.010239 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":205,"server-negotiator.queue_time_us":227,"spinlock_wait_cycles":3968,"thread_start_us":148,"threads_started":1}
I20260926 07:30:18.010977 29399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:17.998017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:44679 (local address 127.0.0.1:44706)
0926 07:30:17.998675 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:17.998753 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:17.999478 (+ 725us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:17.999897 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:17.999903 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:18.000352 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:18.000678 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:18.000700 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:18.001594 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:18.001600 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:18.002489 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:18.002498 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:18.002979 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:18.002985 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:18.003102 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:18.003858 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:18.003875 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:18.005479 (+ 1604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:30:18.007687 (+ 2208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:18.007695 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:18.007705 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:18.008057 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:18.008397 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:18.008400 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:18.008402 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:18.008635 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:18.009179 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:30:18.009185 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:30:18.009459 (+ 274us) client_negotiation.cc:769] Sending connection context
0926 07:30:18.009651 (+ 192us) client_negotiation.cc:240] Negotiation successful
0926 07:30:18.010049 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"mutex_wait_us":54,"spinlock_wait_cycles":7168,"thread_start_us":156,"threads_started":1}
I20260926 07:30:18.012503 29396 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.53.253" port: 45889 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 07:30:18 dist-test-slave-wdjc krb5kdc[28819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407817, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.53.253@KRBTEST.COM for oryx/127.22.53.253@KRBTEST.COM
I20260926 07:30:18.020601 29399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:18.013109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.253:45889 (local address 127.0.0.1:34540)
0926 07:30:18.013281 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:18.013295 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:18.013367 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:18.013583 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:18.013585 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:18.013730 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:18.013822 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:18.013827 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:18.014861 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:18.014864 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:18.015391 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:18.015398 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:18.015796 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:18.015798 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:18.015869 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:18.016311 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:18.016321 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:18.017707 (+ 1386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:30:18.019312 (+ 1605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:18.019314 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:18.019316 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:18.019497 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:18.019721 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:18.019723 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:18.019724 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:18.019792 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:18.020106 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:30:18.020108 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:30:18.020203 (+ 95us) client_negotiation.cc:769] Sending connection context
0926 07:30:18.020283 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 07:30:18.020388 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20260926 07:30:18.020627 29400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:18.013277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34540 (local address 127.22.53.253:45889)
0926 07:30:18.013377 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:18.013381 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:18.013394 (+ 13us) server_negotiation.cc:407] Connection header received
0926 07:30:18.013432 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:18.013436 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:18.013479 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:18.013558 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:18.013917 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:18.014768 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:18.015485 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:18.015702 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:18.015776 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:18.017836 (+ 2060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:18.017838 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:18.017840 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:18.017861 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:18.019202 (+ 1341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:18.019577 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:18.019580 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:18.019581 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:18.019646 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:18.019863 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:18.019865 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:18.019866 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:18.020032 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:18.020104 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:18.020336 (+ 232us) server_negotiation.cc:299] Negotiation successful
0926 07:30:18.020435 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":40}
I20260926 07:30:18.023736 29396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5343811b11a34d0483a88e2adef7fc12: Bootstrap starting.
I20260926 07:30:18.026366 29396 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5343811b11a34d0483a88e2adef7fc12: Neither blocks nor log segments found. Creating new log.
I20260926 07:30:18.027173 29396 log.cc:824] T 00000000000000000000000000000000 P 5343811b11a34d0483a88e2adef7fc12: Log is configured to *not* fsync() on all Append() calls
I20260926 07:30:18.029349 29396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5343811b11a34d0483a88e2adef7fc12: No bootstrap required, opened a new log
I20260926 07:30:18.032493 29396 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5343811b11a34d0483a88e2adef7fc12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b71a0534077f44f7b2cd8a7fa7688789" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 44679 } } peers { permanent_uuid: "5343811b11a34d0483a88e2adef7fc12" member_type: VOTER last_known_addr { host: "127.22.53.253" port: 45889 } }
I20260926 07:30:18.032817 29396 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5343811b11a34d0483a88e2adef7fc12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:30:18.032907 29396 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5343811b11a34d0483a88e2adef7fc12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5343811b11a34d0483a88e2adef7fc12, State: Initialized, Role: FOLLOWER
I20260926 07:30:18.033396 29396 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5343811b11a34d0483a88e2adef7fc12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b71a0534077f44f7b2cd8a7fa7688789" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 44679 } } peers { permanent_uuid: "5343811b11a34d0483a88e2adef7fc12" member_type: VOTER last_known_addr { host: "127.22.53.253" port: 45889 } }
I20260926 07:30:18.033962 29402 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5343811b11a34d0483a88e2adef7fc12 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b71a0534077f44f7b2cd8a7fa7688789" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 44679 } } peers { permanent_uuid: "5343811b11a34d0483a88e2adef7fc12" member_type: VOTER last_known_addr { host: "127.22.53.253" port: 45889 } } }
I20260926 07:30:18.034090 29402 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5343811b11a34d0483a88e2adef7fc12 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:30:18.034735 29396 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5343811b11a34d0483a88e2adef7fc12 [sys.catalog]: configured and running, proceeding with master startup.
W20260926 07:30:18.039847 29413 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5343811b11a34d0483a88e2adef7fc12: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:30:18.039960 29413 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:30:18.042338 29328 master_runner.cc:428] Detected that this master 5343811b11a34d0483a88e2adef7fc12 is joining an existing cluster
I20260926 07:30:18.042457 29328 master_runner.cc:432] Initiating AddMaster RPC to add 127.22.53.253:45889
I20260926 07:30:18.054090 29420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:18.045208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.253:45889 (local address 127.0.0.1:34548)
0926 07:30:18.045531 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:18.045546 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:18.045666 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:18.045980 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:18.045986 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:18.046194 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:18.046335 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:18.046343 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:18.047074 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:18.047081 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:18.048033 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:18.048043 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:18.048946 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:18.048952 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:18.049037 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:18.049718 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:18.049733 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:18.050160 (+ 427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:30:18.052294 (+ 2134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:18.052298 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:18.052301 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:18.052695 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:18.052972 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:18.052975 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:18.052977 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:18.053080 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:18.053510 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:30:18.053513 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:30:18.053610 (+ 97us) client_negotiation.cc:769] Sending connection context
0926 07:30:18.053721 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 07:30:18.053877 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":78,"threads_started":1}
I20260926 07:30:18.054123 28903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:18.044826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44708 (local address 127.22.53.254:44679)
0926 07:30:18.044974 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:18.044979 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:18.045203 (+ 224us) server_negotiation.cc:407] Connection header received
0926 07:30:18.045417 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:18.045421 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:18.045472 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:18.045524 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:18.046326 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:18.047176 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:18.048338 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:18.048662 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:18.048780 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:18.050205 (+ 1425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:18.050209 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:18.050211 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:18.050242 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:18.052243 (+ 2001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:18.052783 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:18.052787 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:18.052790 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:18.052884 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:18.053222 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:18.053227 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:18.053231 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:18.053428 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:18.053501 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:18.053856 (+ 355us) server_negotiation.cc:299] Negotiation successful
0926 07:30:18.053977 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 07:30:18.054123 29400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:18.045275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34548 (local address 127.22.53.253:45889)
0926 07:30:18.045387 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:18.045391 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:18.045638 (+ 247us) server_negotiation.cc:407] Connection header received
0926 07:30:18.045737 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:18.045742 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:18.045789 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:18.045869 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:18.046448 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:18.046963 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:18.048173 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:18.048656 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:18.048787 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:18.050290 (+ 1503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:18.050296 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:18.050301 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:18.050322 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:18.052167 (+ 1845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:18.052808 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:18.052811 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:18.052812 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:18.052881 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:18.053181 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:18.053186 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:18.053190 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:18.053409 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:18.053507 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:18.053789 (+ 282us) server_negotiation.cc:299] Negotiation successful
0926 07:30:18.053950 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":80,"server-negotiator.queue_time_us":41}
I20260926 07:30:18.054586 29419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:18.044742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:44679 (local address 127.0.0.1:44708)
0926 07:30:18.045090 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:18.045103 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:18.045288 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:18.045697 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:18.045700 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:18.046011 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:18.046181 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:18.046189 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:18.047307 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:18.047310 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:18.048218 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:18.048232 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:18.048788 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:18.048792 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:18.048890 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:18.049602 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:18.049620 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:18.050046 (+ 426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:30:18.052331 (+ 2285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:18.052337 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:18.052340 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:18.052681 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:18.052986 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:18.052989 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:18.052991 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:18.053095 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:18.053580 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:30:18.053583 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:30:18.053658 (+ 75us) client_negotiation.cc:769] Sending connection context
0926 07:30:18.054267 (+ 609us) client_negotiation.cc:240] Negotiation successful
0926 07:30:18.054405 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"mutex_wait_us":1,"thread_start_us":119,"threads_started":1}
Sep 26 07:30:18 dist-test-slave-wdjc krb5kdc[28819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407816, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.53.254@KRBTEST.COM for oryx/127.22.53.253@KRBTEST.COM
I20260926 07:30:18.070508 29400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:18.059563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34562 (local address 127.22.53.253:45889)
0926 07:30:18.059723 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:18.059732 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:18.060184 (+ 452us) server_negotiation.cc:407] Connection header received
0926 07:30:18.060391 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:18.060393 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:18.060444 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:18.060523 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:18.061377 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:18.061905 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:18.063191 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:18.063417 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:18.063511 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:18.066550 (+ 3039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:18.066554 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:18.066557 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:18.066587 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:18.068348 (+ 1761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:18.069003 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:18.069006 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:18.069007 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:18.069100 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:18.069458 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:18.069461 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:18.069462 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:18.069723 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:18.069809 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:18.070277 (+ 468us) server_negotiation.cc:299] Negotiation successful
0926 07:30:18.070386 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 07:30:18.070675 29421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:18.059629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.253:45889 (local address 127.0.0.1:34562)
0926 07:30:18.060034 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:18.060087 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:18.060242 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:18.060597 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:18.060603 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:18.061065 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:18.061214 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:18.061222 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:18.062024 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:18.062027 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:18.063029 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:18.063041 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:18.063541 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:18.063545 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:18.063664 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:18.064332 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:18.064344 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:18.066398 (+ 2054us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:30:18.068503 (+ 2105us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:18.068514 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:18.068522 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:18.068872 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:18.069222 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:18.069225 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:18.069227 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:18.069325 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:18.069845 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:30:18.069853 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:30:18.070013 (+ 160us) client_negotiation.cc:769] Sending connection context
0926 07:30:18.070207 (+ 194us) client_negotiation.cc:240] Negotiation successful
0926 07:30:18.070463 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":121,"threads_started":1}
I20260926 07:30:18.072757 28864 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.22.53.253:45889: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "5343811b11a34d0483a88e2adef7fc12"
member_type: NON_VOTER
last_known_addr {
host: "127.22.53.253"
port: 45889
}
attrs {
promote: true
}
}
dest_uuid: "b71a0534077f44f7b2cd8a7fa7688789"
cas_config_opid_index: -1
I20260926 07:30:18.073607 28864 consensus_queue.cc:237] T 00000000000000000000000000000000 P b71a0534077f44f7b2cd8a7fa7688789 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b71a0534077f44f7b2cd8a7fa7688789" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 44679 } } peers { permanent_uuid: "5343811b11a34d0483a88e2adef7fc12" member_type: NON_VOTER last_known_addr { host: "127.22.53.253" port: 45889 } attrs { promote: true } }
WARNING: no policy specified for oryx/127.22.53.252@KRBTEST.COM; defaulting to no policy
I20260926 07:30:18.076179 29424 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b71a0534077f44f7b2cd8a7fa7688789 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 5343811b11a34d0483a88e2adef7fc12 (127.22.53.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b71a0534077f44f7b2cd8a7fa7688789" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 44679 } } peers { permanent_uuid: "5343811b11a34d0483a88e2adef7fc12" member_type: NON_VOTER last_known_addr { host: "127.22.53.253" port: 45889 } attrs { promote: true } } }
I20260926 07:30:18.077103 29370 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5343811b11a34d0483a88e2adef7fc12 [term 0 FOLLOWER]: Advancing to term 1
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.53.252@KRBTEST.COM" created.
I20260926 07:30:18.078227 29424 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.22.53.253:45889
I20260926 07:30:18.078662 29370 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5343811b11a34d0483a88e2adef7fc12 [term 1 FOLLOWER]: Refusing update from remote peer b71a0534077f44f7b2cd8a7fa7688789: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260926 07:30:18.078884 29424 sys_catalog.cc:455] T 00000000000000000000000000000000 P b71a0534077f44f7b2cd8a7fa7688789 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "b71a0534077f44f7b2cd8a7fa7688789" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b71a0534077f44f7b2cd8a7fa7688789" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 44679 } } peers { permanent_uuid: "5343811b11a34d0483a88e2adef7fc12" member_type: NON_VOTER last_known_addr { host: "127.22.53.253" port: 45889 } attrs { promote: true } } }
I20260926 07:30:18.078982 29424 sys_catalog.cc:458] T 00000000000000000000000000000000 P b71a0534077f44f7b2cd8a7fa7688789 [sys.catalog]: This master's current role is: LEADER
I20260926 07:30:18.079358 29328 master.cc:537] Master@127.22.53.253:45889 shutting down...
I20260926 07:30:18.079582 29425 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b71a0534077f44f7b2cd8a7fa7688789 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5343811b11a34d0483a88e2adef7fc12" member_type: NON_VOTER last_known_addr { host: "127.22.53.253" port: 45889 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 07:30:18.081853 28848 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260926 07:30:18.082898 28848 consensus_peers.cc:487] T 00000000000000000000000000000000 P b71a0534077f44f7b2cd8a7fa7688789 -> Peer 5343811b11a34d0483a88e2adef7fc12 (127.22.53.253:45889): Couldn't send request to peer 5343811b11a34d0483a88e2adef7fc12. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 07:30:18.087040 29328 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 5343811b11a34d0483a88e2adef7fc12 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:30:18.087380 29328 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 5343811b11a34d0483a88e2adef7fc12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:30:18.087495 29328 tablet_replica.cc:330] T 00000000000000000000000000000000 P 5343811b11a34d0483a88e2adef7fc12: stopping tablet replica
I20260926 07:30:18.092330 29328 master.cc:552] Master@127.22.53.253:45889 shutdown complete.
I20260926 07:30:18.092396 29328 master_runner.cc:305] Clearing existing system tablet
I20260926 07:30:18.092671 29328 fs_manager.cc:359] Metadata directory not provided
I20260926 07:30:18.092729 29328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:30:18.094130 29328 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 07:30:18.095113 29430 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:18.095422 29328 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:30:18.095569 29328 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.53.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/krb5kdc/oryx_127.22.53.252.keytab.
Entry for principal oryx/127.22.53.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/krb5kdc/oryx_127.22.53.252.keytab.
I20260926 07:30:18.095608 29328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/master-1/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/master-1/wal
uuid: "5343811b11a34d0483a88e2adef7fc12"
format_stamp: "Formatted at 2026-09-26 07:30:17 on dist-test-slave-wdjc"
server_key: "604acb043a909af244ce155ab1e927d3"
server_key_iv: "6e8511f4e42c03e9983acc3bcb248031"
server_key_version: "encryptionkey@0"
I20260926 07:30:18.096732 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.53.252:39937
--webserver_interface=127.22.53.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/krb5kdc/oryx_127.22.53.252.keytab
--principal=oryx/127.22.53.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:40785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.53.254:44679,127.22.53.253:45889,127.22.53.252:39937
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:30:18.097880 29328 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 5343811b11a34d0483a88e2adef7fc12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:30:18.100515 29328 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 5343811b11a34d0483a88e2adef7fc12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 07:30:18.100697 29328 log.cc:1196] T 00000000000000000000000000000000 P 5343811b11a34d0483a88e2adef7fc12: Deleting WAL directory at /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260926 07:30:18.100966 29328 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 5343811b11a34d0483a88e2adef7fc12: Deleting consensus metadata
I20260926 07:30:18.101156 29328 master_runner.cc:315] Copying system tablet from 127.22.53.254:44679
I20260926 07:30:18.102363 29328 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 5343811b11a34d0483a88e2adef7fc12: tablet copy: Beginning tablet copy session from remote peer at address 127.22.53.254:44679
I20260926 07:30:18.110870 28903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:18.103034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44722 (local address 127.22.53.254:44679)
0926 07:30:18.103172 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:18.103180 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:18.103395 (+ 215us) server_negotiation.cc:407] Connection header received
0926 07:30:18.103547 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:18.103549 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:18.103604 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:18.103693 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:18.104149 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:18.104789 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:18.105601 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:18.105832 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:18.105946 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:18.107539 (+ 1593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:18.107543 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:18.107546 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:18.107574 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:18.109099 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:18.109589 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:18.109593 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:18.109595 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:18.109680 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:18.109966 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:18.109971 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:18.109972 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:18.110168 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:18.110288 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:18.110569 (+ 281us) server_negotiation.cc:299] Negotiation successful
0926 07:30:18.110682 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 07:30:18.110888 29437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:18.102976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:44679 (local address 127.0.0.1:44722)
0926 07:30:18.103299 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:18.103311 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:18.103445 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:18.103716 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:18.103719 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:18.103930 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:18.104028 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:18.104034 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:18.104902 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:18.104906 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:18.105475 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:18.105483 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:18.105953 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:18.105956 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:18.106052 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:18.107024 (+ 972us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:18.107036 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:18.107382 (+ 346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:30:18.109261 (+ 1879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:18.109266 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:18.109268 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:18.109482 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:18.109776 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:18.109780 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:18.109782 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:18.109870 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:18.110275 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:30:18.110278 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:30:18.110389 (+ 111us) client_negotiation.cc:769] Sending connection context
0926 07:30:18.110515 (+ 126us) client_negotiation.cc:240] Negotiation successful
0926 07:30:18.110682 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":87,"threads_started":1}
I20260926 07:30:18.111953 28884 tablet_copy_service.cc:140] P b71a0534077f44f7b2cd8a7fa7688789: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 5343811b11a34d0483a88e2adef7fc12 ({username='oryx', principal='oryx/127.22.53.253@KRBTEST.COM'} at 127.0.0.1:44722)
I20260926 07:30:18.112156 28884 tablet_copy_service.cc:161] P b71a0534077f44f7b2cd8a7fa7688789: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 5343811b11a34d0483a88e2adef7fc12 at {username='oryx', principal='oryx/127.22.53.253@KRBTEST.COM'} at 127.0.0.1:44722: session id = 5343811b11a34d0483a88e2adef7fc12-00000000000000000000000000000000
I20260926 07:30:18.113734 28884 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P b71a0534077f44f7b2cd8a7fa7688789: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 07:30:18.115358 29328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 07:30:18.117885 29328 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 5343811b11a34d0483a88e2adef7fc12: tablet copy: Starting download of 0 data blocks...
I20260926 07:30:18.118158 29328 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 5343811b11a34d0483a88e2adef7fc12: tablet copy: Starting download of 1 WAL segments...
I20260926 07:30:18.120260 29328 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 5343811b11a34d0483a88e2adef7fc12: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 07:30:18.122264 28884 tablet_copy_service.cc:342] P b71a0534077f44f7b2cd8a7fa7688789: Request end of tablet copy session 5343811b11a34d0483a88e2adef7fc12-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.22.53.253@KRBTEST.COM'} at 127.0.0.1:44722
I20260926 07:30:18.122409 28884 tablet_copy_service.cc:433] P b71a0534077f44f7b2cd8a7fa7688789: ending tablet copy session 5343811b11a34d0483a88e2adef7fc12-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 5343811b11a34d0483a88e2adef7fc12
I20260926 07:30:18.128937 29328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:30:18.130831 29438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:18.133731 29439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:18.134030 29328 server_base.cc:1034] running on GCE node
W20260926 07:30:18.134051 29441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:18.134369 29328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:30:18.134730 29328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:30:18.135932 29328 hybrid_clock.cc:648] HybridClock initialized: now 1790407818135919 us; error 31 us; skew 500 ppm
Sep 26 07:30:18 dist-test-slave-wdjc krb5kdc[28819](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407818, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.53.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:30:18.137641 29328 init.cc:377] Logged in from keytab as oryx/127.22.53.253@KRBTEST.COM (short username oryx)
I20260926 07:30:18.138149 29328 webserver.cc:460] Webserver started at http://127.22.53.253:44005/ using document root <none> and password file <none>
I20260926 07:30:18.138347 29328 fs_manager.cc:359] Metadata directory not provided
I20260926 07:30:18.138403 29328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:30:18.139667 29328 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 07:30:18.140678 29448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:18.140933 29328 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 07:30:18.141026 29328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/master-1/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/master-1/wal
uuid: "5343811b11a34d0483a88e2adef7fc12"
format_stamp: "Formatted at 2026-09-26 07:30:17 on dist-test-slave-wdjc"
server_key: "604acb043a909af244ce155ab1e927d3"
server_key_iv: "6e8511f4e42c03e9983acc3bcb248031"
server_key_version: "encryptionkey@0"
I20260926 07:30:18.141136 29328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:30:18.153494 29328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:30:18.155673 29328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:30:18.160753 29328 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.253:45889
I20260926 07:30:18.160982 29500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.253:45889 every 8 connection(s)
I20260926 07:30:18.161438 29328 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/master-1/data/info.pb
I20260926 07:30:18.163048 29501 sys_catalog.cc:263] Verifying existing consensus state
I20260926 07:30:18.163913 29501 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5343811b11a34d0483a88e2adef7fc12: Bootstrap starting.
I20260926 07:30:18.172458 29501 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5343811b11a34d0483a88e2adef7fc12: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 07:30:18.172895 29501 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5343811b11a34d0483a88e2adef7fc12: Bootstrap complete.
I20260926 07:30:18.173498 29501 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5343811b11a34d0483a88e2adef7fc12 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b71a0534077f44f7b2cd8a7fa7688789" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 44679 } } peers { permanent_uuid: "5343811b11a34d0483a88e2adef7fc12" member_type: NON_VOTER last_known_addr { host: "127.22.53.253" port: 45889 } attrs { promote: true } }
I20260926 07:30:18.173646 29501 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5343811b11a34d0483a88e2adef7fc12 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5343811b11a34d0483a88e2adef7fc12, State: Initialized, Role: LEARNER
I20260926 07:30:18.173831 29501 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5343811b11a34d0483a88e2adef7fc12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b71a0534077f44f7b2cd8a7fa7688789" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 44679 } } peers { permanent_uuid: "5343811b11a34d0483a88e2adef7fc12" member_type: NON_VOTER last_known_addr { host: "127.22.53.253" port: 45889 } attrs { promote: true } }
I20260926 07:30:18.174199 29504 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5343811b11a34d0483a88e2adef7fc12 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b71a0534077f44f7b2cd8a7fa7688789" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 44679 } } peers { permanent_uuid: "5343811b11a34d0483a88e2adef7fc12" member_type: NON_VOTER last_known_addr { host: "127.22.53.253" port: 45889 } attrs { promote: true } } }
I20260926 07:30:18.174405 29501 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5343811b11a34d0483a88e2adef7fc12 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:30:18.174422 29504 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5343811b11a34d0483a88e2adef7fc12 [sys.catalog]: This master's current role is: LEARNER
I20260926 07:30:18.178100 29515 catalog_manager.cc:1259] Loaded cluster ID: 8a78e9ea9d25425092fd72f238f444f5
I20260926 07:30:18.178185 29515 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5343811b11a34d0483a88e2adef7fc12: loading cluster ID for follower catalog manager: success
I20260926 07:30:18.179811 29515 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5343811b11a34d0483a88e2adef7fc12: acquiring CA information for follower catalog manager: success
I20260926 07:30:18.181111 29515 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5343811b11a34d0483a88e2adef7fc12: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260926 07:30:18.263501 29431 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:30:18.263782 29431 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:30:18.263824 29431 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:30:18.268046 29431 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:30:18.268126 29431 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:30:18.268146 29431 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:30:18.268162 29431 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:30:18.268178 29431 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:30:18.273887 29431 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:40785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.22.53.254:44679,127.22.53.253:45889,127.22.53.252:39937
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/krb5kdc/oryx_127.22.53.252.keytab
--principal=oryx/127.22.53.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.53.252:39937
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.22.53.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29431
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:30:18.275194 29431 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:30:18.276363 29431 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:30:18.285310 29520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:18.285274 29521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:18.285282 29523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:18.285507 29431 server_base.cc:1034] running on GCE node
I20260926 07:30:18.286265 29431 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:30:18.287607 29431 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:30:18.288800 29431 hybrid_clock.cc:648] HybridClock initialized: now 1790407818288780 us; error 43 us; skew 500 ppm
Sep 26 07:30:18 dist-test-slave-wdjc krb5kdc[28819](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407818, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.53.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:30:18.291889 29431 init.cc:377] Logged in from keytab as oryx/127.22.53.252@KRBTEST.COM (short username oryx)
I20260926 07:30:18.292974 29431 webserver.cc:460] Webserver started at http://127.22.53.252:42997/ using document root <none> and password file <none>
I20260926 07:30:18.293622 29431 fs_manager.cc:359] Metadata directory not provided
I20260926 07:30:18.293712 29431 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:30:18.293987 29431 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:30:18.296244 29431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/master-2/data/instance:
uuid: "07cfc39c8b0a4037bc133186d8f0c5aa"
format_stamp: "Formatted at 2026-09-26 07:30:18 on dist-test-slave-wdjc"
server_key: "530544f61fb8ea07bfc89f5bf5092895"
server_key_iv: "24542c428b43632257f35e8338b95410"
server_key_version: "encryptionkey@0"
I20260926 07:30:18.296880 29431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/master-2/wal/instance:
uuid: "07cfc39c8b0a4037bc133186d8f0c5aa"
format_stamp: "Formatted at 2026-09-26 07:30:18 on dist-test-slave-wdjc"
server_key: "530544f61fb8ea07bfc89f5bf5092895"
server_key_iv: "24542c428b43632257f35e8338b95410"
server_key_version: "encryptionkey@0"
I20260926 07:30:18.300838 29431 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 07:30:18.303413 29530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:18.304878 29431 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:30:18.305034 29431 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/master-2/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/master-2/wal
uuid: "07cfc39c8b0a4037bc133186d8f0c5aa"
format_stamp: "Formatted at 2026-09-26 07:30:18 on dist-test-slave-wdjc"
server_key: "530544f61fb8ea07bfc89f5bf5092895"
server_key_iv: "24542c428b43632257f35e8338b95410"
server_key_version: "encryptionkey@0"
I20260926 07:30:18.305164 29431 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:30:18.317991 29055 heartbeater.cc:503] Master 127.22.53.254:44679 was elected leader, sending a full tablet report...
I20260926 07:30:18.330956 29431 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:30:18.334386 29431 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:30:18.334632 29431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:30:18.342700 29431 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.252:39937
I20260926 07:30:18.343012 29582 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.252:39937 every 8 connection(s)
I20260926 07:30:18.343779 29431 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/master-2/data/info.pb
I20260926 07:30:18.346796 29583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:30:18.350320 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 29431
I20260926 07:30:18.350850 29583 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.53.254" port: 44679 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 07:30:18 dist-test-slave-wdjc krb5kdc[28819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407818, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.53.252@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM
Sep 26 07:30:18 dist-test-slave-wdjc krb5kdc[28819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407816, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.22.53.252@KRBTEST.COM
I20260926 07:30:18.365078 28903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:18.352988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44732 (local address 127.22.53.254:44679)
0926 07:30:18.353121 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:18.353125 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:18.353348 (+ 223us) server_negotiation.cc:407] Connection header received
0926 07:30:18.353745 (+ 397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:18.353751 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:18.353810 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:18.353887 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:18.354981 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:18.355813 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:18.357254 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:18.357599 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:18.357718 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:18.360867 (+ 3149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:18.360873 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:18.360877 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:18.360918 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:18.362655 (+ 1737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:18.363230 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:18.363235 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:18.363237 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:18.363335 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:18.363746 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:18.363749 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:18.363751 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:18.364015 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:18.364115 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:18.364736 (+ 621us) server_negotiation.cc:299] Negotiation successful
0926 07:30:18.364893 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 07:30:18.365995 29585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:18.352001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56396 (local address 127.22.53.252:39937)
0926 07:30:18.352458 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:18.352472 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:18.352506 (+ 34us) server_negotiation.cc:407] Connection header received
0926 07:30:18.353431 (+ 925us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:18.353451 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:18.353654 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:18.353877 (+ 223us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:18.354826 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:18.356087 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:18.357254 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:18.357845 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:18.358020 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:18.361292 (+ 3272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:18.361299 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:18.361312 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:18.361334 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:18.363254 (+ 1920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:18.363766 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:18.363772 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:18.363773 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:18.363847 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:18.364132 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:18.364135 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:18.364137 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:18.364430 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:18.364659 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:18.364911 (+ 252us) server_negotiation.cc:299] Negotiation successful
0926 07:30:18.365093 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":107,"server-negotiator.queue_time_us":313,"thread_start_us":122,"threads_started":1}
I20260926 07:30:18.365994 29586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:18.352776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:44679 (local address 127.0.0.1:44732)
0926 07:30:18.353198 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:18.353247 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:18.353538 (+ 291us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:18.353960 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:18.353966 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:18.354446 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:18.354842 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:18.354855 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:18.355941 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:18.355943 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:18.357111 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:18.357119 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:18.357768 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:18.357771 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:18.357986 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:18.358781 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:18.358805 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:18.360650 (+ 1845us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:30:18.362768 (+ 2118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:18.362776 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:18.362787 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:18.363113 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:18.363426 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:18.363430 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:18.363435 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:18.363621 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:18.364119 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:30:18.364127 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:30:18.364447 (+ 320us) client_negotiation.cc:769] Sending connection context
0926 07:30:18.364677 (+ 230us) client_negotiation.cc:240] Negotiation successful
0926 07:30:18.365038 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"mutex_wait_us":22,"thread_start_us":177,"threads_started":1}
I20260926 07:30:18.367375 29583 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.53.253" port: 45889 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 07:30:18 dist-test-slave-wdjc krb5kdc[28819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407818, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.53.252@KRBTEST.COM for oryx/127.22.53.253@KRBTEST.COM
I20260926 07:30:18.377358 29586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:18.367930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.253:45889 (local address 127.0.0.1:34566)
0926 07:30:18.368054 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:18.368069 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:18.368147 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:18.368646 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:18.368649 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:18.368842 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:18.368939 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:18.368945 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:18.369899 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:18.369904 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:18.370462 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:18.370471 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:18.370956 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:18.370959 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:18.371062 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:18.371614 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:18.371627 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:18.373098 (+ 1471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:30:18.375371 (+ 2273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:18.375374 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:18.375377 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:18.375704 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:18.376173 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:18.376176 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:18.376178 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:18.376325 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:18.376873 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:30:18.376876 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:30:18.376973 (+ 97us) client_negotiation.cc:769] Sending connection context
0926 07:30:18.377070 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 07:30:18.377211 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260926 07:30:18.377381 29587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:18.368055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34566 (local address 127.22.53.253:45889)
0926 07:30:18.368363 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:18.368368 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:18.368382 (+ 14us) server_negotiation.cc:407] Connection header received
0926 07:30:18.368425 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:18.368427 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:18.368477 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:18.368635 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:18.369067 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:18.369785 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:18.370595 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:18.370837 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:18.370951 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:18.373296 (+ 2345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:18.373300 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:18.373302 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:18.373329 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:18.375221 (+ 1892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:18.375914 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:18.375918 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:18.375922 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:18.376029 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:18.376472 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:18.376476 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:18.376478 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:18.376779 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:18.376870 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:18.377129 (+ 259us) server_negotiation.cc:299] Negotiation successful
0926 07:30:18.377256 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":111,"threads_started":1}
I20260926 07:30:18.378185 29583 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.53.252" port: 39937 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 07:30:18 dist-test-slave-wdjc krb5kdc[28819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407818, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.53.252@KRBTEST.COM for oryx/127.22.53.252@KRBTEST.COM
I20260926 07:30:18.386053 29585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:18.378746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56404 (local address 127.22.53.252:39937)
0926 07:30:18.378857 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:18.378861 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:18.378872 (+ 11us) server_negotiation.cc:407] Connection header received
0926 07:30:18.378906 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:18.378908 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:18.378941 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:18.379003 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:18.379354 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:18.379847 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:18.380610 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:18.380836 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:18.380914 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:18.382998 (+ 2084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:18.383001 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:18.383003 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:18.383024 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:18.384324 (+ 1300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:18.384870 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:18.384873 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:18.384874 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:18.384941 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:18.385252 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:18.385254 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:18.385255 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:18.385423 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:18.385493 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:18.385796 (+ 303us) server_negotiation.cc:299] Negotiation successful
0926 07:30:18.385926 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":46}
I20260926 07:30:18.386059 29586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:18.378640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.252:39937 (local address 127.0.0.1:56404)
0926 07:30:18.378763 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:18.378778 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:18.378854 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:18.379035 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:18.379037 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:18.379180 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:18.379258 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:18.379262 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:18.379954 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:18.379956 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:18.380457 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:18.380464 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:18.380952 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:18.380955 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:18.381016 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:18.381638 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:18.381660 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:18.382859 (+ 1199us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:30:18.384445 (+ 1586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:18.384448 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:18.384450 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:18.384780 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:18.385051 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:18.385054 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:18.385055 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:18.385150 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:18.385530 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:30:18.385533 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:30:18.385649 (+ 116us) client_negotiation.cc:769] Sending connection context
0926 07:30:18.385746 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 07:30:18.385870 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260926 07:30:18.389564 29583 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07cfc39c8b0a4037bc133186d8f0c5aa: Bootstrap starting.
I20260926 07:30:18.391930 29583 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 07cfc39c8b0a4037bc133186d8f0c5aa: Neither blocks nor log segments found. Creating new log.
I20260926 07:30:18.392765 29583 log.cc:824] T 00000000000000000000000000000000 P 07cfc39c8b0a4037bc133186d8f0c5aa: Log is configured to *not* fsync() on all Append() calls
I20260926 07:30:18.395079 29583 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07cfc39c8b0a4037bc133186d8f0c5aa: No bootstrap required, opened a new log
I20260926 07:30:18.398456 29583 raft_consensus.cc:348] T 00000000000000000000000000000000 P 07cfc39c8b0a4037bc133186d8f0c5aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b71a0534077f44f7b2cd8a7fa7688789" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 44679 } } peers { permanent_uuid: "5343811b11a34d0483a88e2adef7fc12" member_type: VOTER last_known_addr { host: "127.22.53.253" port: 45889 } } peers { permanent_uuid: "07cfc39c8b0a4037bc133186d8f0c5aa" member_type: VOTER last_known_addr { host: "127.22.53.252" port: 39937 } }
I20260926 07:30:18.398687 29583 raft_consensus.cc:374] T 00000000000000000000000000000000 P 07cfc39c8b0a4037bc133186d8f0c5aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:30:18.398730 29583 raft_consensus.cc:729] T 00000000000000000000000000000000 P 07cfc39c8b0a4037bc133186d8f0c5aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07cfc39c8b0a4037bc133186d8f0c5aa, State: Initialized, Role: FOLLOWER
I20260926 07:30:18.399186 29583 consensus_queue.cc:260] T 00000000000000000000000000000000 P 07cfc39c8b0a4037bc133186d8f0c5aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b71a0534077f44f7b2cd8a7fa7688789" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 44679 } } peers { permanent_uuid: "5343811b11a34d0483a88e2adef7fc12" member_type: VOTER last_known_addr { host: "127.22.53.253" port: 45889 } } peers { permanent_uuid: "07cfc39c8b0a4037bc133186d8f0c5aa" member_type: VOTER last_known_addr { host: "127.22.53.252" port: 39937 } }
I20260926 07:30:18.399690 29589 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07cfc39c8b0a4037bc133186d8f0c5aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b71a0534077f44f7b2cd8a7fa7688789" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 44679 } } peers { permanent_uuid: "5343811b11a34d0483a88e2adef7fc12" member_type: VOTER last_known_addr { host: "127.22.53.253" port: 45889 } } peers { permanent_uuid: "07cfc39c8b0a4037bc133186d8f0c5aa" member_type: VOTER last_known_addr { host: "127.22.53.252" port: 39937 } } }
I20260926 07:30:18.399849 29589 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07cfc39c8b0a4037bc133186d8f0c5aa [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:30:18.400643 29583 sys_catalog.cc:564] T 00000000000000000000000000000000 P 07cfc39c8b0a4037bc133186d8f0c5aa [sys.catalog]: configured and running, proceeding with master startup.
W20260926 07:30:18.405992 29600 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 07cfc39c8b0a4037bc133186d8f0c5aa: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:30:18.406126 29600 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:30:18.409679 29431 master_runner.cc:428] Detected that this master 07cfc39c8b0a4037bc133186d8f0c5aa is joining an existing cluster
I20260926 07:30:18.409773 29431 master_runner.cc:432] Initiating AddMaster RPC to add 127.22.53.252:39937
I20260926 07:30:18.419984 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 28926
I20260926 07:30:18.427855 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.53.193:35569
--local_ip_for_outbound_sockets=127.22.53.193
--tserver_master_addrs=127.22.53.254:44679
--webserver_port=40931
--webserver_interface=127.22.53.193
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/krb5kdc/oryx_127.22.53.193.keytab
--principal=oryx/127.22.53.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:40785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.22.53.254:44679,127.22.53.253:45889
--tserver_master_addrs=127.22.53.254:44679,127.22.53.253:45889,127.22.53.252:39937 with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:30:18.436507 29608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:18.412508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.252:39937 (local address 127.0.0.1:56416)
0926 07:30:18.413243 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:18.413257 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:18.413377 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:18.413721 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:18.413725 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:18.413994 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:18.414131 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:18.414139 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:18.415144 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:18.415147 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:18.416067 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:18.416077 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:18.416803 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:18.416806 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:18.416899 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:18.417456 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:18.417473 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:18.417986 (+ 513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:30:18.434558 (+ 16572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:18.434565 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:18.434569 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:18.434856 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:18.435168 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:18.435172 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:18.435176 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:18.435267 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:18.435797 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:30:18.435801 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:30:18.435912 (+ 111us) client_negotiation.cc:769] Sending connection context
0926 07:30:18.436031 (+ 119us) client_negotiation.cc:240] Negotiation successful
0926 07:30:18.436214 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":671,"thread_start_us":77,"threads_started":1}
I20260926 07:30:18.437192 29585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:18.412381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56416 (local address 127.22.53.252:39937)
0926 07:30:18.412511 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:18.412515 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:18.413322 (+ 807us) server_negotiation.cc:407] Connection header received
0926 07:30:18.413473 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:18.413476 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:18.413515 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:18.413563 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:18.414254 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:18.415024 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:18.416188 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:18.416517 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:18.416630 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:18.418184 (+ 1554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:18.418188 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:18.418191 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:18.418216 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:18.434416 (+ 16200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:18.434972 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:18.434976 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:18.434977 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:18.435061 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:18.435368 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:18.435374 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:18.435378 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:18.435638 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:18.435755 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:18.436808 (+ 1053us) server_negotiation.cc:299] Negotiation successful
0926 07:30:18.436943 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 07:30:18.439229 28903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:18.413319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44734 (local address 127.22.53.254:44679)
0926 07:30:18.413497 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:18.413504 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:18.413517 (+ 13us) server_negotiation.cc:407] Connection header received
0926 07:30:18.413562 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:18.413565 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:18.413616 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:18.413697 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:18.414422 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:18.415243 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:18.416578 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:18.416927 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:18.416995 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:18.434971 (+ 17976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:18.434975 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:18.434980 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:18.435023 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:18.436821 (+ 1798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:18.437497 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:18.437504 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:18.437509 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:18.437621 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:18.438013 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:18.438019 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:18.438023 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:18.438302 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:18.438391 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:18.438843 (+ 452us) server_negotiation.cc:299] Negotiation successful
0926 07:30:18.438995 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 07:30:18.439790 29587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:18.413032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34576 (local address 127.22.53.253:45889)
0926 07:30:18.413187 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:18.413200 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:18.413485 (+ 285us) server_negotiation.cc:407] Connection header received
0926 07:30:18.413693 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:18.413696 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:18.413775 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:18.413830 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:18.414430 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:18.415243 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:18.416869 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:18.417542 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:18.417606 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:18.433737 (+ 16131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:18.433746 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:18.433749 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:18.433787 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:18.435614 (+ 1827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:18.436959 (+ 1345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:18.436963 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:18.436965 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:18.437067 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:18.438771 (+ 1704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:18.438776 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:18.438777 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:18.439035 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:18.439130 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:18.439493 (+ 363us) server_negotiation.cc:299] Negotiation successful
0926 07:30:18.439643 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 07:30:18.439944 29606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:18.412383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:44679 (local address 127.0.0.1:44734)
0926 07:30:18.412739 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:18.412754 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:18.412877 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:18.413818 (+ 941us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:18.413822 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:18.414130 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:18.414269 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:18.414278 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:18.415390 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:18.415393 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:18.416412 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:18.416422 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:18.417157 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:18.417160 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:18.417279 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:18.433455 (+ 16176us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:18.433481 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:18.434551 (+ 1070us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:30:18.436961 (+ 2410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:18.436964 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:18.436966 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:18.437239 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:18.437770 (+ 531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:18.437775 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:18.437779 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:18.437886 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:18.438477 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:30:18.438483 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:30:18.438604 (+ 121us) client_negotiation.cc:769] Sending connection context
0926 07:30:18.439474 (+ 870us) client_negotiation.cc:240] Negotiation successful
0926 07:30:18.439665 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"mutex_wait_us":23,"thread_start_us":145,"threads_started":1}
I20260926 07:30:18.440382 29607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:18.412944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.253:45889 (local address 127.0.0.1:34576)
0926 07:30:18.413386 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:18.413401 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:18.413575 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:18.413983 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:18.413989 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:18.414193 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:18.414301 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:18.414311 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:18.415390 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:18.415393 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:18.416748 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:18.416757 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:18.417740 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:18.417742 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:18.417808 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:18.418355 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:18.418373 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:18.433371 (+ 14998us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:30:18.435761 (+ 2390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:18.435766 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:18.435768 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:18.436034 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:18.437202 (+ 1168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:18.437206 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:18.437209 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:18.437553 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:18.439190 (+ 1637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:30:18.439196 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:30:18.439285 (+ 89us) client_negotiation.cc:769] Sending connection context
0926 07:30:18.439989 (+ 704us) client_negotiation.cc:240] Negotiation successful
0926 07:30:18.440200 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"mutex_wait_us":3,"thread_start_us":101,"threads_started":1}
Sep 26 07:30:18 dist-test-slave-wdjc krb5kdc[28819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407816, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.53.254@KRBTEST.COM for oryx/127.22.53.252@KRBTEST.COM
I20260926 07:30:18.454778 29421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:18.444751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.252:39937 (local address 127.0.0.1:56424)
0926 07:30:18.444937 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:18.444952 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:18.445050 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:18.445373 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:18.445377 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:18.445599 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:18.445740 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:18.445747 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:18.446827 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:18.446831 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:18.447705 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:18.447718 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:18.448336 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:18.448340 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:18.448445 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:18.449196 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:18.449220 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:18.450884 (+ 1664us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:30:18.453056 (+ 2172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:18.453061 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:18.453063 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:18.453339 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:18.453646 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:18.453649 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:18.453652 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:18.453759 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:18.454185 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:30:18.454188 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:30:18.454312 (+ 124us) client_negotiation.cc:769] Sending connection context
0926 07:30:18.454420 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 07:30:18.454591 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110}
I20260926 07:30:18.455158 29585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:18.444713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56424 (local address 127.22.53.252:39937)
0926 07:30:18.444841 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:18.444846 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:18.445027 (+ 181us) server_negotiation.cc:407] Connection header received
0926 07:30:18.445170 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:18.445173 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:18.445244 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:18.445334 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:18.445868 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:18.446660 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:18.447844 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:18.448200 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:18.448312 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:18.451057 (+ 2745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:18.451061 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:18.451065 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:18.451093 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:18.452928 (+ 1835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:18.453447 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:18.453451 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:18.453453 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:18.453546 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:18.453854 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:18.453857 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:18.453859 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:18.454086 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:18.454184 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:18.454500 (+ 316us) server_negotiation.cc:299] Negotiation successful
0926 07:30:18.454621 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 07:30:18.458994 28864 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.22.53.252:39937: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "07cfc39c8b0a4037bc133186d8f0c5aa"
member_type: NON_VOTER
last_known_addr {
host: "127.22.53.252"
port: 39937
}
attrs {
promote: true
}
}
dest_uuid: "b71a0534077f44f7b2cd8a7fa7688789"
cas_config_opid_index: 5
I20260926 07:30:18.459722 28864 consensus_queue.cc:237] T 00000000000000000000000000000000 P b71a0534077f44f7b2cd8a7fa7688789 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b71a0534077f44f7b2cd8a7fa7688789" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 44679 } } peers { permanent_uuid: "5343811b11a34d0483a88e2adef7fc12" member_type: NON_VOTER last_known_addr { host: "127.22.53.253" port: 45889 } attrs { promote: true } } peers { permanent_uuid: "07cfc39c8b0a4037bc133186d8f0c5aa" member_type: NON_VOTER last_known_addr { host: "127.22.53.252" port: 39937 } attrs { promote: true } }
I20260926 07:30:18.461000 29424 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b71a0534077f44f7b2cd8a7fa7688789 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 07cfc39c8b0a4037bc133186d8f0c5aa (127.22.53.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b71a0534077f44f7b2cd8a7fa7688789" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 44679 } } peers { permanent_uuid: "5343811b11a34d0483a88e2adef7fc12" member_type: NON_VOTER last_known_addr { host: "127.22.53.253" port: 45889 } attrs { promote: true } } peers { permanent_uuid: "07cfc39c8b0a4037bc133186d8f0c5aa" member_type: NON_VOTER last_known_addr { host: "127.22.53.252" port: 39937 } attrs { promote: true } } }
I20260926 07:30:18.462463 29424 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.22.53.252:39937
I20260926 07:30:18.462647 29424 sys_catalog.cc:455] T 00000000000000000000000000000000 P b71a0534077f44f7b2cd8a7fa7688789 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "b71a0534077f44f7b2cd8a7fa7688789" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b71a0534077f44f7b2cd8a7fa7688789" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 44679 } } peers { permanent_uuid: "5343811b11a34d0483a88e2adef7fc12" member_type: NON_VOTER last_known_addr { host: "127.22.53.253" port: 45889 } attrs { promote: true } } peers { permanent_uuid: "07cfc39c8b0a4037bc133186d8f0c5aa" member_type: NON_VOTER last_known_addr { host: "127.22.53.252" port: 39937 } attrs { promote: true } } }
I20260926 07:30:18.462754 29424 sys_catalog.cc:458] T 00000000000000000000000000000000 P b71a0534077f44f7b2cd8a7fa7688789 [sys.catalog]: This master's current role is: LEADER
I20260926 07:30:18.463730 29557 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 07cfc39c8b0a4037bc133186d8f0c5aa [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:30:18.463831 29431 master.cc:537] Master@127.22.53.252:39937 shutting down...
I20260926 07:30:18.465435 29557 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 07cfc39c8b0a4037bc133186d8f0c5aa [term 1 FOLLOWER]: Refusing update from remote peer b71a0534077f44f7b2cd8a7fa7688789: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 07:30:18.466486 29425 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b71a0534077f44f7b2cd8a7fa7688789 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07cfc39c8b0a4037bc133186d8f0c5aa" member_type: NON_VOTER last_known_addr { host: "127.22.53.252" port: 39937 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 07:30:18.468772 28849 consensus_peers.cc:487] T 00000000000000000000000000000000 P b71a0534077f44f7b2cd8a7fa7688789 -> Peer 07cfc39c8b0a4037bc133186d8f0c5aa (127.22.53.252:39937): Couldn't send request to peer 07cfc39c8b0a4037bc133186d8f0c5aa. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 07:30:18.469147 29587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:18.461443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34590 (local address 127.22.53.253:45889)
0926 07:30:18.461561 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:18.461566 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:18.461717 (+ 151us) server_negotiation.cc:407] Connection header received
0926 07:30:18.461908 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:18.461912 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:18.461974 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:18.462039 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:30:18.462453 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:18.463252 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:18.467288 (+ 4036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:18.467997 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:18.468083 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:18.468251 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:18.468900 (+ 649us) server_negotiation.cc:299] Negotiation successful
0926 07:30:18.469003 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 07:30:18.469578 29421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:18.461172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.253:45889 (local address 127.0.0.1:34590)
0926 07:30:18.461651 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:18.461665 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:18.461789 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:18.462216 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:18.462219 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:18.462236 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:30:18.462328 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:18.462334 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:18.465546 (+ 3212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:18.465549 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:18.467096 (+ 1547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:18.467114 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:18.468356 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:18.468359 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:18.468467 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:18.468524 (+ 57us) client_negotiation.cc:769] Sending connection context
0926 07:30:18.469359 (+ 835us) client_negotiation.cc:240] Negotiation successful
0926 07:30:18.469457 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":409}
I20260926 07:30:18.470134 29475 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5343811b11a34d0483a88e2adef7fc12 [term 1 LEARNER]: Refusing update from remote peer b71a0534077f44f7b2cd8a7fa7688789: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 07:30:18.470376 29431 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 07cfc39c8b0a4037bc133186d8f0c5aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:30:18.470592 29425 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b71a0534077f44f7b2cd8a7fa7688789 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5343811b11a34d0483a88e2adef7fc12" member_type: NON_VOTER last_known_addr { host: "127.22.53.253" port: 45889 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260926 07:30:18.470705 29431 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 07cfc39c8b0a4037bc133186d8f0c5aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:30:18.470815 29431 tablet_replica.cc:330] T 00000000000000000000000000000000 P 07cfc39c8b0a4037bc133186d8f0c5aa: stopping tablet replica
I20260926 07:30:18.472671 29475 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5343811b11a34d0483a88e2adef7fc12 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 07cfc39c8b0a4037bc133186d8f0c5aa (127.22.53.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b71a0534077f44f7b2cd8a7fa7688789" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 44679 } } peers { permanent_uuid: "5343811b11a34d0483a88e2adef7fc12" member_type: NON_VOTER last_known_addr { host: "127.22.53.253" port: 45889 } attrs { promote: true } } peers { permanent_uuid: "07cfc39c8b0a4037bc133186d8f0c5aa" member_type: NON_VOTER last_known_addr { host: "127.22.53.252" port: 39937 } attrs { promote: true } } }
I20260926 07:30:18.476121 29431 master.cc:552] Master@127.22.53.252:39937 shutdown complete.
I20260926 07:30:18.476176 29431 master_runner.cc:305] Clearing existing system tablet
I20260926 07:30:18.476227 29504 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5343811b11a34d0483a88e2adef7fc12 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b71a0534077f44f7b2cd8a7fa7688789. Latest consensus state: current_term: 1 leader_uuid: "b71a0534077f44f7b2cd8a7fa7688789" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b71a0534077f44f7b2cd8a7fa7688789" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 44679 } } peers { permanent_uuid: "5343811b11a34d0483a88e2adef7fc12" member_type: NON_VOTER last_known_addr { host: "127.22.53.253" port: 45889 } attrs { promote: true } } peers { permanent_uuid: "07cfc39c8b0a4037bc133186d8f0c5aa" member_type: NON_VOTER last_known_addr { host: "127.22.53.252" port: 39937 } attrs { promote: true } } }
I20260926 07:30:18.476352 29504 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5343811b11a34d0483a88e2adef7fc12 [sys.catalog]: This master's current role is: LEARNER
I20260926 07:30:18.476416 29431 fs_manager.cc:359] Metadata directory not provided
I20260926 07:30:18.476464 29431 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:30:18.476442 29504 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5343811b11a34d0483a88e2adef7fc12 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b71a0534077f44f7b2cd8a7fa7688789" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b71a0534077f44f7b2cd8a7fa7688789" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 44679 } } peers { permanent_uuid: "5343811b11a34d0483a88e2adef7fc12" member_type: NON_VOTER last_known_addr { host: "127.22.53.253" port: 45889 } attrs { promote: true } } peers { permanent_uuid: "07cfc39c8b0a4037bc133186d8f0c5aa" member_type: NON_VOTER last_known_addr { host: "127.22.53.252" port: 39937 } attrs { promote: true } } }
I20260926 07:30:18.476503 29504 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5343811b11a34d0483a88e2adef7fc12 [sys.catalog]: This master's current role is: LEARNER
I20260926 07:30:18.477454 29425 raft_consensus.cc:1050] T 00000000000000000000000000000000 P b71a0534077f44f7b2cd8a7fa7688789: attempting to promote NON_VOTER 5343811b11a34d0483a88e2adef7fc12 to VOTER
I20260926 07:30:18.477953 29425 consensus_queue.cc:237] T 00000000000000000000000000000000 P b71a0534077f44f7b2cd8a7fa7688789 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b71a0534077f44f7b2cd8a7fa7688789" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 44679 } } peers { permanent_uuid: "5343811b11a34d0483a88e2adef7fc12" member_type: VOTER last_known_addr { host: "127.22.53.253" port: 45889 } attrs { promote: false } } peers { permanent_uuid: "07cfc39c8b0a4037bc133186d8f0c5aa" member_type: NON_VOTER last_known_addr { host: "127.22.53.252" port: 39937 } attrs { promote: true } }
I20260926 07:30:18.479382 29475 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5343811b11a34d0483a88e2adef7fc12 [term 1 LEARNER]: Refusing update from remote peer b71a0534077f44f7b2cd8a7fa7688789: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260926 07:30:18.479449 29421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:18.479116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.252:39937 (local address 127.0.0.1:56432)
0926 07:30:18.479279 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:18.479356 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.252:39937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260926 07:30:18.479880 29424 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b71a0534077f44f7b2cd8a7fa7688789 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5343811b11a34d0483a88e2adef7fc12" member_type: VOTER last_known_addr { host: "127.22.53.253" port: 45889 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260926 07:30:18.480393 29421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:18.479954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.252:39937 (local address 127.0.0.1:56436)
0926 07:30:18.480235 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:18.480298 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.252:39937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":85,"threads_started":1}
W20260926 07:30:18.480479 28849 consensus_peers.cc:487] T 00000000000000000000000000000000 P b71a0534077f44f7b2cd8a7fa7688789 -> Peer 07cfc39c8b0a4037bc133186d8f0c5aa (127.22.53.252:39937): Couldn't send request to peer 07cfc39c8b0a4037bc133186d8f0c5aa. Status: Network error: Client connection negotiation failed: client connection to 127.22.53.252:39937: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 07:30:18.481619 29431 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.001s sys 0.000s
I20260926 07:30:18.482383 29424 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b71a0534077f44f7b2cd8a7fa7688789 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 5343811b11a34d0483a88e2adef7fc12 (127.22.53.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b71a0534077f44f7b2cd8a7fa7688789" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 44679 } } peers { permanent_uuid: "5343811b11a34d0483a88e2adef7fc12" member_type: VOTER last_known_addr { host: "127.22.53.253" port: 45889 } attrs { promote: false } } peers { permanent_uuid: "07cfc39c8b0a4037bc133186d8f0c5aa" member_type: NON_VOTER last_known_addr { host: "127.22.53.252" port: 39937 } attrs { promote: true } } }
I20260926 07:30:18.482606 29615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:18.482797 29431 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:30:18.482904 29431 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 07:30:18.482937 29431 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/master-2/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/master-2/wal
uuid: "07cfc39c8b0a4037bc133186d8f0c5aa"
format_stamp: "Formatted at 2026-09-26 07:30:18 on dist-test-slave-wdjc"
server_key: "530544f61fb8ea07bfc89f5bf5092895"
server_key_iv: "24542c428b43632257f35e8338b95410"
server_key_version: "encryptionkey@0"
I20260926 07:30:18.483901 29424 sys_catalog.cc:455] T 00000000000000000000000000000000 P b71a0534077f44f7b2cd8a7fa7688789 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "b71a0534077f44f7b2cd8a7fa7688789" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b71a0534077f44f7b2cd8a7fa7688789" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 44679 } } peers { permanent_uuid: "5343811b11a34d0483a88e2adef7fc12" member_type: VOTER last_known_addr { host: "127.22.53.253" port: 45889 } attrs { promote: false } } peers { permanent_uuid: "07cfc39c8b0a4037bc133186d8f0c5aa" member_type: NON_VOTER last_known_addr { host: "127.22.53.252" port: 39937 } attrs { promote: true } } }
I20260926 07:30:18.484010 29424 sys_catalog.cc:458] T 00000000000000000000000000000000 P b71a0534077f44f7b2cd8a7fa7688789 [sys.catalog]: This master's current role is: LEADER
I20260926 07:30:18.484122 29424 sys_catalog.cc:455] T 00000000000000000000000000000000 P b71a0534077f44f7b2cd8a7fa7688789 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b71a0534077f44f7b2cd8a7fa7688789" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b71a0534077f44f7b2cd8a7fa7688789" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 44679 } } peers { permanent_uuid: "5343811b11a34d0483a88e2adef7fc12" member_type: VOTER last_known_addr { host: "127.22.53.253" port: 45889 } attrs { promote: false } } peers { permanent_uuid: "07cfc39c8b0a4037bc133186d8f0c5aa" member_type: NON_VOTER last_known_addr { host: "127.22.53.252" port: 39937 } attrs { promote: true } } }
I20260926 07:30:18.484185 29424 sys_catalog.cc:458] T 00000000000000000000000000000000 P b71a0534077f44f7b2cd8a7fa7688789 [sys.catalog]: This master's current role is: LEADER
I20260926 07:30:18.485155 29431 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 07cfc39c8b0a4037bc133186d8f0c5aa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:30:18.485188 29474 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5343811b11a34d0483a88e2adef7fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 5343811b11a34d0483a88e2adef7fc12 (127.22.53.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b71a0534077f44f7b2cd8a7fa7688789" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 44679 } } peers { permanent_uuid: "5343811b11a34d0483a88e2adef7fc12" member_type: VOTER last_known_addr { host: "127.22.53.253" port: 45889 } attrs { promote: false } } peers { permanent_uuid: "07cfc39c8b0a4037bc133186d8f0c5aa" member_type: NON_VOTER last_known_addr { host: "127.22.53.252" port: 39937 } attrs { promote: true } } }
I20260926 07:30:18.486964 29504 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5343811b11a34d0483a88e2adef7fc12 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b71a0534077f44f7b2cd8a7fa7688789" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b71a0534077f44f7b2cd8a7fa7688789" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 44679 } } peers { permanent_uuid: "5343811b11a34d0483a88e2adef7fc12" member_type: VOTER last_known_addr { host: "127.22.53.253" port: 45889 } attrs { promote: false } } peers { permanent_uuid: "07cfc39c8b0a4037bc133186d8f0c5aa" member_type: NON_VOTER last_known_addr { host: "127.22.53.252" port: 39937 } attrs { promote: true } } }
I20260926 07:30:18.487075 29504 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5343811b11a34d0483a88e2adef7fc12 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:30:18.487725 29431 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 07cfc39c8b0a4037bc133186d8f0c5aa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 07:30:18.487819 29431 log.cc:1196] T 00000000000000000000000000000000 P 07cfc39c8b0a4037bc133186d8f0c5aa: Deleting WAL directory at /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260926 07:30:18.488054 29431 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 07cfc39c8b0a4037bc133186d8f0c5aa: Deleting consensus metadata
I20260926 07:30:18.488215 29431 master_runner.cc:315] Copying system tablet from 127.22.53.254:44679
I20260926 07:30:18.489446 29431 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 07cfc39c8b0a4037bc133186d8f0c5aa: tablet copy: Beginning tablet copy session from remote peer at address 127.22.53.254:44679
I20260926 07:30:18.499476 28903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:18.490362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44736 (local address 127.22.53.254:44679)
0926 07:30:18.490507 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:18.490512 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:18.490529 (+ 17us) server_negotiation.cc:407] Connection header received
0926 07:30:18.490687 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:18.490690 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:18.490741 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:18.490827 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:18.491345 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:18.492160 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:18.493468 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:18.493839 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:18.493951 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:18.495414 (+ 1463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:18.495419 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:18.495422 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:18.495452 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:18.497336 (+ 1884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:18.497911 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:18.497915 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:18.497917 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:18.498013 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:18.498398 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:18.498402 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:18.498403 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:18.498649 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:18.498757 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:18.499154 (+ 397us) server_negotiation.cc:299] Negotiation successful
0926 07:30:18.499306 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 07:30:18.499539 29621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:18.489997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:44679 (local address 127.0.0.1:44736)
0926 07:30:18.490444 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:18.490461 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:18.490583 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:18.490869 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:18.490872 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:18.491103 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:18.491221 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:18.491229 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:18.492304 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:18.492310 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:18.493318 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:18.493330 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:18.493973 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:18.493977 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:18.494082 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:18.494726 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:18.494742 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:18.495242 (+ 500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:30:18.497471 (+ 2229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:18.497475 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:18.497478 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:18.497776 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:18.498153 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:18.498157 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:18.498161 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:18.498279 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:18.498782 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:30:18.498787 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:30:18.498934 (+ 147us) client_negotiation.cc:769] Sending connection context
0926 07:30:18.499075 (+ 141us) client_negotiation.cc:240] Negotiation successful
0926 07:30:18.499287 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":97,"threads_started":1}
I20260926 07:30:18.500324 28884 tablet_copy_service.cc:140] P b71a0534077f44f7b2cd8a7fa7688789: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 07cfc39c8b0a4037bc133186d8f0c5aa ({username='oryx', principal='oryx/127.22.53.252@KRBTEST.COM'} at 127.0.0.1:44736)
I20260926 07:30:18.500427 28884 tablet_copy_service.cc:161] P b71a0534077f44f7b2cd8a7fa7688789: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 07cfc39c8b0a4037bc133186d8f0c5aa at {username='oryx', principal='oryx/127.22.53.252@KRBTEST.COM'} at 127.0.0.1:44736: session id = 07cfc39c8b0a4037bc133186d8f0c5aa-00000000000000000000000000000000
I20260926 07:30:18.501631 28884 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P b71a0534077f44f7b2cd8a7fa7688789: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 07:30:18.503037 29431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 07:30:18.505738 29431 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 07cfc39c8b0a4037bc133186d8f0c5aa: tablet copy: Starting download of 0 data blocks...
I20260926 07:30:18.505949 29431 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 07cfc39c8b0a4037bc133186d8f0c5aa: tablet copy: Starting download of 1 WAL segments...
I20260926 07:30:18.507697 29431 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 07cfc39c8b0a4037bc133186d8f0c5aa: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 07:30:18.510046 28884 tablet_copy_service.cc:342] P b71a0534077f44f7b2cd8a7fa7688789: Request end of tablet copy session 07cfc39c8b0a4037bc133186d8f0c5aa-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.22.53.252@KRBTEST.COM'} at 127.0.0.1:44736
I20260926 07:30:18.510159 28884 tablet_copy_service.cc:433] P b71a0534077f44f7b2cd8a7fa7688789: ending tablet copy session 07cfc39c8b0a4037bc133186d8f0c5aa-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 07cfc39c8b0a4037bc133186d8f0c5aa
I20260926 07:30:18.518198 29431 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:30:18.520244 29622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:18.520267 29625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:18.521026 29623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:18.524646 29431 server_base.cc:1034] running on GCE node
I20260926 07:30:18.525170 29431 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:30:18.525532 29431 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:30:18.526691 29431 hybrid_clock.cc:648] HybridClock initialized: now 1790407818526663 us; error 59 us; skew 500 ppm
Sep 26 07:30:18 dist-test-slave-wdjc krb5kdc[28819](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407818, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.53.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:30:18.528420 29431 init.cc:377] Logged in from keytab as oryx/127.22.53.252@KRBTEST.COM (short username oryx)
I20260926 07:30:18.529014 29431 webserver.cc:460] Webserver started at http://127.22.53.252:41175/ using document root <none> and password file <none>
I20260926 07:30:18.529210 29431 fs_manager.cc:359] Metadata directory not provided
I20260926 07:30:18.529253 29431 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:30:18.530695 29431 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 07:30:18.531752 29632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:18.532037 29431 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 07:30:18.532124 29431 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/master-2/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/master-2/wal
uuid: "07cfc39c8b0a4037bc133186d8f0c5aa"
format_stamp: "Formatted at 2026-09-26 07:30:18 on dist-test-slave-wdjc"
server_key: "530544f61fb8ea07bfc89f5bf5092895"
server_key_iv: "24542c428b43632257f35e8338b95410"
server_key_version: "encryptionkey@0"
I20260926 07:30:18.532212 29431 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:30:18.546142 29189 heartbeater.cc:503] Master 127.22.53.254:44679 was elected leader, sending a full tablet report...
I20260926 07:30:18.555128 29431 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:30:18.557502 29431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:30:18.562718 29431 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.252:39937
I20260926 07:30:18.562757 29684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.252:39937 every 8 connection(s)
I20260926 07:30:18.563372 29431 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/master-2/data/info.pb
I20260926 07:30:18.565459 29685 sys_catalog.cc:263] Verifying existing consensus state
I20260926 07:30:18.566699 29685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07cfc39c8b0a4037bc133186d8f0c5aa: Bootstrap starting.
I20260926 07:30:18.580790 29685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07cfc39c8b0a4037bc133186d8f0c5aa: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 07:30:18.581202 29685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07cfc39c8b0a4037bc133186d8f0c5aa: Bootstrap complete.
I20260926 07:30:18.581956 29685 raft_consensus.cc:348] T 00000000000000000000000000000000 P 07cfc39c8b0a4037bc133186d8f0c5aa [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b71a0534077f44f7b2cd8a7fa7688789" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 44679 } } peers { permanent_uuid: "5343811b11a34d0483a88e2adef7fc12" member_type: VOTER last_known_addr { host: "127.22.53.253" port: 45889 } attrs { promote: false } } peers { permanent_uuid: "07cfc39c8b0a4037bc133186d8f0c5aa" member_type: NON_VOTER last_known_addr { host: "127.22.53.252" port: 39937 } attrs { promote: true } }
I20260926 07:30:18.582149 29685 raft_consensus.cc:729] T 00000000000000000000000000000000 P 07cfc39c8b0a4037bc133186d8f0c5aa [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 07cfc39c8b0a4037bc133186d8f0c5aa, State: Initialized, Role: LEARNER
I20260926 07:30:18.582289 29685 consensus_queue.cc:260] T 00000000000000000000000000000000 P 07cfc39c8b0a4037bc133186d8f0c5aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b71a0534077f44f7b2cd8a7fa7688789" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 44679 } } peers { permanent_uuid: "5343811b11a34d0483a88e2adef7fc12" member_type: VOTER last_known_addr { host: "127.22.53.253" port: 45889 } attrs { promote: false } } peers { permanent_uuid: "07cfc39c8b0a4037bc133186d8f0c5aa" member_type: NON_VOTER last_known_addr { host: "127.22.53.252" port: 39937 } attrs { promote: true } }
I20260926 07:30:18.582719 29688 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07cfc39c8b0a4037bc133186d8f0c5aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b71a0534077f44f7b2cd8a7fa7688789" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 44679 } } peers { permanent_uuid: "5343811b11a34d0483a88e2adef7fc12" member_type: VOTER last_known_addr { host: "127.22.53.253" port: 45889 } attrs { promote: false } } peers { permanent_uuid: "07cfc39c8b0a4037bc133186d8f0c5aa" member_type: NON_VOTER last_known_addr { host: "127.22.53.252" port: 39937 } attrs { promote: true } } }
I20260926 07:30:18.582888 29685 sys_catalog.cc:564] T 00000000000000000000000000000000 P 07cfc39c8b0a4037bc133186d8f0c5aa [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:30:18.582921 29688 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07cfc39c8b0a4037bc133186d8f0c5aa [sys.catalog]: This master's current role is: LEARNER
I20260926 07:30:18.587239 29699 catalog_manager.cc:1259] Loaded cluster ID: 8a78e9ea9d25425092fd72f238f444f5
I20260926 07:30:18.587318 29699 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 07cfc39c8b0a4037bc133186d8f0c5aa: loading cluster ID for follower catalog manager: success
I20260926 07:30:18.589636 29699 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 07cfc39c8b0a4037bc133186d8f0c5aa: acquiring CA information for follower catalog manager: success
I20260926 07:30:18.591125 29699 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 07cfc39c8b0a4037bc133186d8f0c5aa: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260926 07:30:18.594347 29609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:30:18.594604 29609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:30:18.594694 29609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:30:18.599206 29609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:30:18.599275 29609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:30:18.599368 29609 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.53.193
I20260926 07:30:18.605178 29609 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:40785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/krb5kdc/oryx_127.22.53.193.keytab
--principal=oryx/127.22.53.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.53.193:35569
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.53.193
--webserver_port=40931
--tserver_master_addrs=127.22.53.254:44679,127.22.53.253:45889,127.22.53.252:39937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29609
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.53.193
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:30:18.606369 29609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:30:18.607491 29609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:30:18.618678 29705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:18.618818 29609 server_base.cc:1034] running on GCE node
W20260926 07:30:18.618678 29704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:18.618678 29707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:18.619484 29609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:30:18.620275 29609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:30:18.621506 29609 hybrid_clock.cc:648] HybridClock initialized: now 1790407818621472 us; error 58 us; skew 500 ppm
Sep 26 07:30:18 dist-test-slave-wdjc krb5kdc[28819](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407818, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.53.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:30:18.624473 29609 init.cc:377] Logged in from keytab as oryx/127.22.53.193@KRBTEST.COM (short username oryx)
I20260926 07:30:18.625504 29609 webserver.cc:460] Webserver started at http://127.22.53.193:40931/ using document root <none> and password file <none>
I20260926 07:30:18.626099 29609 fs_manager.cc:359] Metadata directory not provided
I20260926 07:30:18.626179 29609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:30:18.630437 29609 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260926 07:30:18.632890 29714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:18.634368 29609 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 07:30:18.634541 29609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/ts-0/wal
uuid: "96b38667cb7946819bcb4dfc480f93be"
format_stamp: "Formatted at 2026-09-26 07:30:17 on dist-test-slave-wdjc"
server_key: "93355256a82da45aebaecc8b8e52aac1"
server_key_iv: "b8dcbaae9d8be84df493735cc9823431"
server_key_version: "encryptionkey@0"
I20260926 07:30:18.635337 29609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:30:18.661580 29609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:30:18.665164 29609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:30:18.665400 29609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:30:18.666078 29609 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:30:18.667009 29609 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:30:18.667085 29609 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:18.667156 29609 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:30:18.667208 29609 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:18.678001 29609 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.193:35569
I20260926 07:30:18.678045 29827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.193:35569 every 8 connection(s)
I20260926 07:30:18.679224 29609 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/ts-0/data/info.pb
I20260926 07:30:18.680214 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 29609
I20260926 07:30:18.680419 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 29060
I20260926 07:30:18.691156 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.53.194:38993
--local_ip_for_outbound_sockets=127.22.53.194
--tserver_master_addrs=127.22.53.254:44679
--webserver_port=37591
--webserver_interface=127.22.53.194
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/krb5kdc/oryx_127.22.53.194.keytab
--principal=oryx/127.22.53.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:40785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.22.53.254:44679,127.22.53.253:45889
--tserver_master_addrs=127.22.53.254:44679,127.22.53.253:45889,127.22.53.252:39937 with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 07:30:18 dist-test-slave-wdjc krb5kdc[28819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407818, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.53.193@KRBTEST.COM for oryx/127.22.53.252@KRBTEST.COM
Sep 26 07:30:18 dist-test-slave-wdjc krb5kdc[28819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407818, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.53.193@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM
Sep 26 07:30:18 dist-test-slave-wdjc krb5kdc[28819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407818, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.53.193@KRBTEST.COM for oryx/127.22.53.253@KRBTEST.COM
I20260926 07:30:18.700874 29834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:18.683082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.193:34635 (local address 127.22.53.252:39937)
0926 07:30:18.683687 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:18.683700 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:18.684278 (+ 578us) server_negotiation.cc:407] Connection header received
0926 07:30:18.685716 (+ 1438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:18.685722 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:18.685801 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:18.685930 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:18.687832 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:18.689192 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:18.691065 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:18.691623 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:18.691757 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:18.695717 (+ 3960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:18.695724 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:18.695730 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:18.695769 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:18.697811 (+ 2042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:18.698672 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:18.698679 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:18.698683 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:18.698795 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:18.699256 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:18.699262 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:18.699266 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:18.699511 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:18.699620 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:18.700462 (+ 842us) server_negotiation.cc:299] Negotiation successful
0926 07:30:18.700640 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":158,"threads_started":1}
I20260926 07:30:18.700884 28903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:18.690319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.193:45715 (local address 127.22.53.254:44679)
0926 07:30:18.690457 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:18.690462 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:18.690475 (+ 13us) server_negotiation.cc:407] Connection header received
0926 07:30:18.690517 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:18.690520 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:18.690565 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:18.690654 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:18.691473 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:18.692310 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:18.693560 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:18.693922 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:18.694011 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:18.696709 (+ 2698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:18.696714 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:18.696719 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:18.696761 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:18.698483 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:18.699069 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:18.699076 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:18.699080 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:18.699200 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:18.699568 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:18.699573 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:18.699575 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:18.699843 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:18.699930 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:18.700441 (+ 511us) server_negotiation.cc:299] Negotiation successful
0926 07:30:18.700646 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 07:30:18.703475 29832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:18.682829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.252:39937 (local address 127.22.53.193:34635)
0926 07:30:18.683992 (+ 1163us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:18.684052 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:18.685417 (+ 1365us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:18.686168 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:18.686180 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:18.686789 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:18.687588 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:18.687614 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:18.689413 (+ 1799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:18.689420 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:18.690920 (+ 1500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:18.690930 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:18.691999 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:18.692004 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:18.692301 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:18.693089 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:18.693117 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:18.695458 (+ 2341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:30:18.697988 (+ 2530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:18.697999 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:18.698026 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:18.698515 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:18.698896 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:18.698900 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:18.698901 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:18.699131 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:18.699699 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:30:18.699710 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:30:18.700102 (+ 392us) client_negotiation.cc:769] Sending connection context
0926 07:30:18.701258 (+ 1156us) client_negotiation.cc:240] Negotiation successful
0926 07:30:18.703045 (+ 1787us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1000,"mutex_wait_us":106,"spinlock_wait_cycles":25856,"thread_start_us":83,"threads_started":1}
I20260926 07:30:18.704028 29835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:18.689604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:44679 (local address 127.22.53.193:45715)
0926 07:30:18.690099 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:18.690114 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:18.690221 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:18.690716 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:18.690721 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:18.690983 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:18.691152 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:18.691159 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:18.692473 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:18.692480 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:18.693387 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:18.693397 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:18.694175 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:18.694179 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:18.694267 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:18.694872 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:18.694887 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:18.696485 (+ 1598us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:30:18.698659 (+ 2174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:18.698663 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:18.698666 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:18.698943 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:18.699330 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:18.699335 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:18.699339 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:18.699442 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:18.700016 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:30:18.700021 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:30:18.700172 (+ 151us) client_negotiation.cc:769] Sending connection context
0926 07:30:18.701464 (+ 1292us) client_negotiation.cc:240] Negotiation successful
0926 07:30:18.701886 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":405,"spinlock_wait_cycles":23680,"thread_start_us":115,"threads_started":1}
I20260926 07:30:18.705119 29828 heartbeater.cc:347] Connected to a master server at 127.22.53.252:39937
I20260926 07:30:18.705284 29587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:18.683762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.193:35855 (local address 127.22.53.253:45889)
0926 07:30:18.683968 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:18.683974 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:18.684184 (+ 210us) server_negotiation.cc:407] Connection header received
0926 07:30:18.693213 (+ 9029us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:18.693217 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:18.693283 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:18.693336 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:18.694048 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:18.695262 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:18.697275 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:18.697703 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:18.697805 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:18.700856 (+ 3051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:18.700860 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:18.700863 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:18.700895 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:18.702776 (+ 1881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:18.703334 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:18.703339 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:18.703341 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:18.703437 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:18.703921 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:18.703927 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:18.703932 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:18.704184 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:18.704297 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:18.704959 (+ 662us) server_negotiation.cc:299] Negotiation successful
0926 07:30:18.705109 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 07:30:18.705425 29828 heartbeater.cc:464] Registering TS with master...
I20260926 07:30:18.705893 29833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:18.681964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.253:45889 (local address 127.22.53.193:35855)
0926 07:30:18.683992 (+ 2028us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:18.684052 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:18.693045 (+ 8993us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:18.693528 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:18.693534 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:18.693751 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:18.693878 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:18.693885 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:18.695435 (+ 1550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:18.695438 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:18.696648 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:18.696656 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:18.698302 (+ 1646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:18.698308 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:18.698431 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:18.699099 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:18.699114 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:18.700684 (+ 1570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:30:18.702938 (+ 2254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:18.702944 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:18.702946 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:18.703196 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:18.703702 (+ 506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:18.703706 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:18.703708 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:18.703802 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:18.704651 (+ 849us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:30:18.704654 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:30:18.704761 (+ 107us) client_negotiation.cc:769] Sending connection context
0926 07:30:18.705606 (+ 845us) client_negotiation.cc:240] Negotiation successful
0926 07:30:18.705768 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1780,"spinlock_wait_cycles":9984,"thread_start_us":734,"threads_started":1}
I20260926 07:30:18.706008 29828 heartbeater.cc:511] Master 127.22.53.252:39937 requested a full tablet report, sending...
I20260926 07:30:18.705116 29829 heartbeater.cc:347] Connected to a master server at 127.22.53.254:44679
I20260926 07:30:18.706458 29829 heartbeater.cc:464] Registering TS with master...
I20260926 07:30:18.706651 29829 heartbeater.cc:511] Master 127.22.53.254:44679 requested a full tablet report, sending...
I20260926 07:30:18.706795 29830 heartbeater.cc:347] Connected to a master server at 127.22.53.253:45889
I20260926 07:30:18.706867 29830 heartbeater.cc:464] Registering TS with master...
I20260926 07:30:18.707028 29830 heartbeater.cc:511] Master 127.22.53.253:45889 requested a full tablet report, sending...
I20260926 07:30:18.707742 29649 ts_manager.cc:194] Registered new tserver with Master: 96b38667cb7946819bcb4dfc480f93be (127.22.53.193:35569)
I20260926 07:30:18.707916 28864 ts_manager.cc:194] Re-registered known tserver with Master: 96b38667cb7946819bcb4dfc480f93be (127.22.53.193:35569)
I20260926 07:30:18.708156 29465 ts_manager.cc:194] Registered new tserver with Master: 96b38667cb7946819bcb4dfc480f93be (127.22.53.193:35569)
I20260926 07:30:18.708726 28864 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.53.193@KRBTEST.COM'} at 127.22.53.193:45715
I20260926 07:30:18.775193 29323 heartbeater.cc:503] Master 127.22.53.254:44679 was elected leader, sending a full tablet report...
W20260926 07:30:18.819064 29836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:30:18.819378 29836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:30:18.819427 29836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:30:18.823689 29836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:30:18.823772 29836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:30:18.823836 29836 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.53.194
I20260926 07:30:18.830248 29836 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:40785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/krb5kdc/oryx_127.22.53.194.keytab
--principal=oryx/127.22.53.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.53.194:38993
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.53.194
--webserver_port=37591
--tserver_master_addrs=127.22.53.254:44679,127.22.53.253:45889,127.22.53.252:39937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29836
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.53.194
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:30:18.831634 29836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:30:18.832983 29836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:30:18.845847 29842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:18.845964 29844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:18.846052 29836 server_base.cc:1034] running on GCE node
W20260926 07:30:18.845995 29841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:18.846938 29836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:30:18.847821 29836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:30:18.849021 29836 hybrid_clock.cc:648] HybridClock initialized: now 1790407818849010 us; error 32 us; skew 500 ppm
Sep 26 07:30:18 dist-test-slave-wdjc krb5kdc[28819](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407818, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.53.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:30:18.852340 29836 init.cc:377] Logged in from keytab as oryx/127.22.53.194@KRBTEST.COM (short username oryx)
I20260926 07:30:18.853478 29836 webserver.cc:460] Webserver started at http://127.22.53.194:37591/ using document root <none> and password file <none>
I20260926 07:30:18.854113 29836 fs_manager.cc:359] Metadata directory not provided
I20260926 07:30:18.854245 29836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:30:18.859066 29836 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 07:30:18.862504 29851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:18.864159 29836 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260926 07:30:18.864390 29836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/ts-1/wal
uuid: "ddbc2223fead4ddfa34d0a137efe01cb"
format_stamp: "Formatted at 2026-09-26 07:30:17 on dist-test-slave-wdjc"
server_key: "67a6896d69677c2372a738d44458e3b3"
server_key_iv: "77be86c71443834c1b7982e75bca4723"
server_key_version: "encryptionkey@0"
I20260926 07:30:18.865187 29836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:30:18.882212 29836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:30:18.886667 29836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:30:18.886948 29836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:30:18.887671 29836 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:30:18.888783 29836 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:30:18.888863 29836 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:18.888967 29836 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:30:18.889011 29836 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:18.899629 29836 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.194:38993
I20260926 07:30:18.899828 29964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.194:38993 every 8 connection(s)
I20260926 07:30:18.901060 29836 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/ts-1/data/info.pb
I20260926 07:30:18.910351 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 29836
I20260926 07:30:18.910555 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 29194
Sep 26 07:30:18 dist-test-slave-wdjc krb5kdc[28819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407818, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.53.194@KRBTEST.COM for oryx/127.22.53.252@KRBTEST.COM
Sep 26 07:30:18 dist-test-slave-wdjc krb5kdc[28819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407818, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.53.194@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM
Sep 26 07:30:18 dist-test-slave-wdjc krb5kdc[28819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407818, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.53.194@KRBTEST.COM for oryx/127.22.53.253@KRBTEST.COM
I20260926 07:30:18.921378 28903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:18.905266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.194:39175 (local address 127.22.53.254:44679)
0926 07:30:18.905407 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:18.905411 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:18.905523 (+ 112us) server_negotiation.cc:407] Connection header received
0926 07:30:18.906332 (+ 809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:18.906335 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:18.906390 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:18.906478 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:18.908008 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:18.908822 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:18.910540 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:18.910919 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:18.911001 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:18.916098 (+ 5097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:18.916104 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:18.916107 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:18.916148 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:18.918227 (+ 2079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:18.919229 (+ 1002us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:18.919234 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:18.919236 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:18.919378 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:18.919847 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:18.919852 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:18.919856 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:18.920134 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:18.920224 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:18.921040 (+ 816us) server_negotiation.cc:299] Negotiation successful
0926 07:30:18.921195 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 07:30:18.921378 29587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:18.903304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.194:36225 (local address 127.22.53.253:45889)
0926 07:30:18.903450 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:18.903455 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:18.904740 (+ 1285us) server_negotiation.cc:407] Connection header received
0926 07:30:18.905940 (+ 1200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:18.905944 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:18.906011 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:18.906121 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:18.908754 (+ 2633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:18.909602 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:18.910913 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:18.911244 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:18.911323 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:18.916550 (+ 5227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:18.916578 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:18.916580 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:18.916617 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:18.919217 (+ 2600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:18.919901 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:18.919908 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:18.919912 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:18.920002 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:18.920324 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:18.920327 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:18.920329 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:18.920581 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:18.920654 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:18.921040 (+ 386us) server_negotiation.cc:299] Negotiation successful
0926 07:30:18.921185 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 07:30:18.921484 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.53.195:42015
--local_ip_for_outbound_sockets=127.22.53.195
--tserver_master_addrs=127.22.53.254:44679
--webserver_port=33573
--webserver_interface=127.22.53.195
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/krb5kdc/oryx_127.22.53.195.keytab
--principal=oryx/127.22.53.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:40785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.22.53.254:44679,127.22.53.253:45889
--tserver_master_addrs=127.22.53.254:44679,127.22.53.253:45889,127.22.53.252:39937 with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:30:18.921531 29834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:18.903254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.194:38719 (local address 127.22.53.252:39937)
0926 07:30:18.903465 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:18.903470 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:18.904557 (+ 1087us) server_negotiation.cc:407] Connection header received
0926 07:30:18.906082 (+ 1525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:18.906084 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:18.906140 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:18.906277 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:18.907831 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:18.908735 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:18.910560 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:18.911061 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:18.911167 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:18.916962 (+ 5795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:18.916970 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:18.916977 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:18.917019 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:18.919208 (+ 2189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:18.919908 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:18.919915 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:18.919920 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:18.920027 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:18.920378 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:18.920385 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:18.920389 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:18.920677 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:18.920770 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:18.921163 (+ 393us) server_negotiation.cc:299] Negotiation successful
0926 07:30:18.921343 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260926 07:30:18.923365 29971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:18.905087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:44679 (local address 127.22.53.194:39175)
0926 07:30:18.905435 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:18.905448 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:18.905682 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:18.906528 (+ 846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:18.906530 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:18.906999 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:18.907883 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:18.907891 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:18.908916 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:18.908921 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:18.910388 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:18.910399 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:18.911241 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:18.911245 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:18.911471 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:18.912457 (+ 986us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:18.912482 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:18.915775 (+ 3293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:30:18.918401 (+ 2626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:18.918413 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:18.918442 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:18.918899 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:18.919502 (+ 603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:18.919506 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:18.919509 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:18.919718 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:18.920316 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:30:18.920324 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:30:18.920735 (+ 411us) client_negotiation.cc:769] Sending connection context
0926 07:30:18.921563 (+ 828us) client_negotiation.cc:240] Negotiation successful
0926 07:30:18.922209 (+ 646us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"mutex_wait_us":362,"spinlock_wait_cycles":49664,"thread_start_us":83,"threads_started":1}
I20260926 07:30:18.923388 29970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:18.903748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.253:45889 (local address 127.22.53.194:36225)
0926 07:30:18.904631 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:18.904645 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:18.905690 (+ 1045us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:18.906340 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:18.906351 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:18.906977 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:18.907649 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:18.907673 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:18.909745 (+ 2072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:18.909749 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:18.910776 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:18.910785 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:18.911482 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:18.911486 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:18.911587 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:18.912457 (+ 870us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:18.912488 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:18.916325 (+ 3837us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:30:18.919366 (+ 3041us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:18.919370 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:18.919372 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:18.919777 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:18.920122 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:18.920125 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:18.920127 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:18.920216 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:18.920737 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:30:18.920740 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:30:18.920828 (+ 88us) client_negotiation.cc:769] Sending connection context
0926 07:30:18.922407 (+ 1579us) client_negotiation.cc:240] Negotiation successful
0926 07:30:18.922559 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":790,"mutex_wait_us":108,"spinlock_wait_cycles":68480,"thread_start_us":177,"threads_started":1}
I20260926 07:30:18.923708 29969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:18.903747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.252:39937 (local address 127.22.53.194:38719)
0926 07:30:18.904345 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:18.904405 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:18.905682 (+ 1277us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:18.906342 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:18.906352 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:18.906977 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:18.907648 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:18.907668 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:18.908909 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:18.908912 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:18.910396 (+ 1484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:18.910408 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:18.911354 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:18.911360 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:18.911501 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:18.912639 (+ 1138us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:18.912661 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:18.916372 (+ 3711us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:30:18.919350 (+ 2978us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:18.919356 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:18.919361 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:18.919777 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:18.920160 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:18.920165 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:18.920170 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:18.920271 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:18.920848 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:30:18.920854 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:30:18.920966 (+ 112us) client_negotiation.cc:769] Sending connection context
0926 07:30:18.921677 (+ 711us) client_negotiation.cc:240] Negotiation successful
0926 07:30:18.922102 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"mutex_wait_us":211,"spinlock_wait_cycles":35200,"thread_start_us":619,"threads_started":1}
I20260926 07:30:18.925148 29965 heartbeater.cc:347] Connected to a master server at 127.22.53.252:39937
I20260926 07:30:18.925154 29967 heartbeater.cc:347] Connected to a master server at 127.22.53.253:45889
I20260926 07:30:18.925432 29965 heartbeater.cc:464] Registering TS with master...
I20260926 07:30:18.925431 29967 heartbeater.cc:464] Registering TS with master...
I20260926 07:30:18.925896 29966 heartbeater.cc:347] Connected to a master server at 127.22.53.254:44679
I20260926 07:30:18.925982 29966 heartbeater.cc:464] Registering TS with master...
I20260926 07:30:18.926162 29966 heartbeater.cc:511] Master 127.22.53.254:44679 requested a full tablet report, sending...
I20260926 07:30:18.926153 29965 heartbeater.cc:511] Master 127.22.53.252:39937 requested a full tablet report, sending...
I20260926 07:30:18.926153 29967 heartbeater.cc:511] Master 127.22.53.253:45889 requested a full tablet report, sending...
I20260926 07:30:18.927657 29649 ts_manager.cc:194] Registered new tserver with Master: ddbc2223fead4ddfa34d0a137efe01cb (127.22.53.194:38993)
I20260926 07:30:18.927758 29465 ts_manager.cc:194] Registered new tserver with Master: ddbc2223fead4ddfa34d0a137efe01cb (127.22.53.194:38993)
I20260926 07:30:18.928467 28864 ts_manager.cc:194] Re-registered known tserver with Master: ddbc2223fead4ddfa34d0a137efe01cb (127.22.53.194:38993)
I20260926 07:30:18.929452 28864 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.53.194@KRBTEST.COM'} at 127.22.53.194:39175
I20260926 07:30:18.941599 29613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:18.937576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.252:39937 (local address 127.0.0.1:56438)
0926 07:30:18.937777 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:18.937794 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:18.937933 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:18.938243 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:18.938245 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:18.938265 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:30:18.938379 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:18.938384 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:18.939275 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:18.939279 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:18.940167 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:18.940180 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:18.941127 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:18.941131 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:18.941228 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:18.941252 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 07:30:18.941343 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 07:30:18.941438 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97}
I20260926 07:30:18.941598 29834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:18.937805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56438 (local address 127.22.53.252:39937)
0926 07:30:18.937954 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:18.937959 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:18.937979 (+ 20us) server_negotiation.cc:407] Connection header received
0926 07:30:18.938042 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:18.938046 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:18.938110 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:18.938220 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:30:18.938501 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:18.939115 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:18.940334 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:18.940978 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:18.941104 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:18.941305 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:18.941361 (+ 56us) server_negotiation.cc:299] Negotiation successful
0926 07:30:18.941437 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 07:30:18.942566 29659 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 07cfc39c8b0a4037bc133186d8f0c5aa [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260926 07:30:18.943022 29688 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07cfc39c8b0a4037bc133186d8f0c5aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader b71a0534077f44f7b2cd8a7fa7688789. Latest consensus state: current_term: 1 leader_uuid: "b71a0534077f44f7b2cd8a7fa7688789" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b71a0534077f44f7b2cd8a7fa7688789" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 44679 } } peers { permanent_uuid: "5343811b11a34d0483a88e2adef7fc12" member_type: VOTER last_known_addr { host: "127.22.53.253" port: 45889 } attrs { promote: false } } peers { permanent_uuid: "07cfc39c8b0a4037bc133186d8f0c5aa" member_type: NON_VOTER last_known_addr { host: "127.22.53.252" port: 39937 } attrs { promote: true } } }
I20260926 07:30:18.943207 29688 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07cfc39c8b0a4037bc133186d8f0c5aa [sys.catalog]: This master's current role is: LEARNER
W20260926 07:30:19.061460 29972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:30:19.061785 29972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:30:19.061836 29972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:30:19.066256 29972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:30:19.066334 29972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:30:19.066397 29972 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.53.195
I20260926 07:30:19.072525 29972 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:40785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/krb5kdc/oryx_127.22.53.195.keytab
--principal=oryx/127.22.53.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.53.195:42015
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.53.195
--webserver_port=33573
--tserver_master_addrs=127.22.53.254:44679,127.22.53.253:45889,127.22.53.252:39937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29972
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.53.195
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:30:19.073848 29972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:30:19.075300 29972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:30:19.087764 29977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:19.087778 29972 server_base.cc:1034] running on GCE node
W20260926 07:30:19.087797 29978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:19.087776 29980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:19.088680 29972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:30:19.089421 29972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:30:19.090637 29972 hybrid_clock.cc:648] HybridClock initialized: now 1790407819090609 us; error 52 us; skew 500 ppm
Sep 26 07:30:19 dist-test-slave-wdjc krb5kdc[28819](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407819, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.53.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:30:19.093827 29972 init.cc:377] Logged in from keytab as oryx/127.22.53.195@KRBTEST.COM (short username oryx)
I20260926 07:30:19.095028 29972 webserver.cc:460] Webserver started at http://127.22.53.195:33573/ using document root <none> and password file <none>
I20260926 07:30:19.095685 29972 fs_manager.cc:359] Metadata directory not provided
I20260926 07:30:19.095814 29972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:30:19.100907 29972 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 07:30:19.103454 29987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:19.105237 29972 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:30:19.105435 29972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/ts-2/wal
uuid: "ebec0858532b452191c80c0652eb9b8e"
format_stamp: "Formatted at 2026-09-26 07:30:17 on dist-test-slave-wdjc"
server_key: "7d3929fcb2ef50584f4326f146df8df2"
server_key_iv: "0463dc008ae7bc0f710c0734c10df298"
server_key_version: "encryptionkey@0"
I20260926 07:30:19.106115 29972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:30:19.117769 29972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:30:19.121312 29972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:30:19.121554 29972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:30:19.122221 29972 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:30:19.123200 29972 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:30:19.123275 29972 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:19.123348 29972 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:30:19.123391 29972 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:19.133590 29972 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.195:42015
I20260926 07:30:19.133684 30100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.195:42015 every 8 connection(s)
I20260926 07:30:19.134795 29972 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/ts-2/data/info.pb
I20260926 07:30:19.141045 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 29972
Sep 26 07:30:19 dist-test-slave-wdjc krb5kdc[28819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407819, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.53.195@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM
Sep 26 07:30:19 dist-test-slave-wdjc krb5kdc[28819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407819, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.53.195@KRBTEST.COM for oryx/127.22.53.253@KRBTEST.COM
Sep 26 07:30:19 dist-test-slave-wdjc krb5kdc[28819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407819, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.53.195@KRBTEST.COM for oryx/127.22.53.252@KRBTEST.COM
I20260926 07:30:19.156697 29834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:19.136805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.195:36723 (local address 127.22.53.252:39937)
0926 07:30:19.137008 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:19.137016 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:19.138058 (+ 1042us) server_negotiation.cc:407] Connection header received
0926 07:30:19.139562 (+ 1504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:19.139565 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:19.139636 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:19.139786 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:19.142025 (+ 2239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:19.142992 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:19.145495 (+ 2503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:19.145847 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:19.145945 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:19.151665 (+ 5720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:19.151670 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:19.151675 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:19.151711 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:19.153778 (+ 2067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:19.154569 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:19.154577 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:19.154580 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:19.154694 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:19.155178 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:19.155184 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:19.155188 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:19.155439 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:19.155522 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:19.156313 (+ 791us) server_negotiation.cc:299] Negotiation successful
0926 07:30:19.156473 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 07:30:19.158036 30106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:19.137364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.252:39937 (local address 127.22.53.195:36723)
0926 07:30:19.137881 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:19.137939 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:19.139294 (+ 1355us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:19.140003 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:19.140008 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:19.140812 (+ 804us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:19.141816 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:19.141839 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:19.143174 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:19.143178 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:19.145354 (+ 2176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:19.145364 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:19.146114 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:19.146117 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:19.146292 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:19.147319 (+ 1027us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:19.147346 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:19.151463 (+ 4117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:30:19.153952 (+ 2489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:19.153961 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:19.153987 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:19.154430 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:19.154822 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:19.154826 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:19.154828 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:19.155057 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:19.155601 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:30:19.155611 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:30:19.156020 (+ 409us) client_negotiation.cc:769] Sending connection context
0926 07:30:19.157202 (+ 1182us) client_negotiation.cc:240] Negotiation successful
0926 07:30:19.157366 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"mutex_wait_us":953,"spinlock_wait_cycles":37632,"thread_start_us":161,"threads_started":1}
I20260926 07:30:19.158252 30112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:19.143868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44742 (local address 127.22.53.254:44679)
0926 07:30:19.145547 (+ 1679us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:19.145551 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:19.146222 (+ 671us) server_negotiation.cc:407] Connection header received
0926 07:30:19.146273 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:19.146276 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:19.146333 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:19.146421 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:19.147647 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:19.148611 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:19.149769 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:19.150075 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:19.150212 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:19.153433 (+ 3221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:19.153438 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:19.153441 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:19.153486 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:19.155642 (+ 2156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:19.156192 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:19.156195 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:19.156197 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:19.156294 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:19.156659 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:19.156665 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:19.156670 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:19.156933 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:19.157077 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:19.157862 (+ 785us) server_negotiation.cc:299] Negotiation successful
0926 07:30:19.158035 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1572,"thread_start_us":90,"threads_started":1}
I20260926 07:30:19.158571 28903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:19.138657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.195:57261 (local address 127.22.53.254:44679)
0926 07:30:19.138794 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:19.138800 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:19.139009 (+ 209us) server_negotiation.cc:407] Connection header received
0926 07:30:19.139714 (+ 705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:19.139717 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:19.139766 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:19.139883 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:19.142034 (+ 2151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:19.142873 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:19.145200 (+ 2327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:19.145534 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:19.145650 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:19.151379 (+ 5729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:19.151383 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:19.151386 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:19.151421 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:19.154898 (+ 3477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:19.155409 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:19.155413 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:19.155415 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:19.155508 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:19.155824 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:19.155827 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:19.155829 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:19.156039 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:19.156149 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:19.158290 (+ 2141us) server_negotiation.cc:299] Negotiation successful
0926 07:30:19.158415 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 07:30:19.159089 30107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:19.138512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:44679 (local address 127.22.53.195:57261)
0926 07:30:19.138905 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:19.138921 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:19.139295 (+ 374us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:19.139997 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:19.140008 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:19.140822 (+ 814us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:19.141825 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:19.141844 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:19.143144 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:19.143146 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:19.145025 (+ 1879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:19.145038 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:19.145852 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:19.145859 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:19.146245 (+ 386us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:19.147503 (+ 1258us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:19.147518 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:19.151166 (+ 3648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:30:19.155017 (+ 3851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:19.155020 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:19.155022 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:19.155293 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:19.155627 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:19.155631 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:19.155632 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:19.155722 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:19.156197 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:30:19.156203 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:30:19.156302 (+ 99us) client_negotiation.cc:769] Sending connection context
0926 07:30:19.156406 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 07:30:19.156837 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"mutex_wait_us":333,"spinlock_wait_cycles":92032,"thread_start_us":115,"threads_started":1}
I20260926 07:30:19.160391 30101 heartbeater.cc:347] Connected to a master server at 127.22.53.252:39937
I20260926 07:30:19.160535 30102 heartbeater.cc:347] Connected to a master server at 127.22.53.254:44679
I20260926 07:30:19.160703 30101 heartbeater.cc:464] Registering TS with master...
I20260926 07:30:19.160701 30102 heartbeater.cc:464] Registering TS with master...
I20260926 07:30:19.161468 30102 heartbeater.cc:511] Master 127.22.53.254:44679 requested a full tablet report, sending...
I20260926 07:30:19.162678 28863 ts_manager.cc:194] Re-registered known tserver with Master: ebec0858532b452191c80c0652eb9b8e (127.22.53.195:42015)
I20260926 07:30:19.163431 28863 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.53.195@KRBTEST.COM'} at 127.22.53.195:57261
I20260926 07:30:19.161468 30101 heartbeater.cc:511] Master 127.22.53.252:39937 requested a full tablet report, sending...
I20260926 07:30:19.164990 29649 ts_manager.cc:194] Registered new tserver with Master: ebec0858532b452191c80c0652eb9b8e (127.22.53.195:42015)
I20260926 07:30:19.161875 28864 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44742:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:30:19.166486 28864 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:30:19.167124 30113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:19.144368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56446 (local address 127.22.53.252:39937)
0926 07:30:19.148481 (+ 4113us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:19.148488 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:19.148509 (+ 21us) server_negotiation.cc:407] Connection header received
0926 07:30:19.148580 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:19.148583 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:19.148640 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:19.148741 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:19.149811 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:19.151064 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:19.153694 (+ 2630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:19.154127 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:19.154244 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:19.156339 (+ 2095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:19.156346 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:19.156351 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:19.156381 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:19.158669 (+ 2288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:19.161682 (+ 3013us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:19.161686 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:19.161688 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:19.161804 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:19.163843 (+ 2039us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:19.163848 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:19.163850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:19.164148 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:19.164320 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:19.166709 (+ 2389us) server_negotiation.cc:299] Negotiation successful
0926 07:30:19.166871 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4001,"thread_start_us":111,"threads_started":1}
I20260926 07:30:19.170332 29612 sys_catalog.cc:455] T 00000000000000000000000000000000 P b71a0534077f44f7b2cd8a7fa7688789 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b71a0534077f44f7b2cd8a7fa7688789" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b71a0534077f44f7b2cd8a7fa7688789" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 44679 } } peers { permanent_uuid: "5343811b11a34d0483a88e2adef7fc12" member_type: VOTER last_known_addr { host: "127.22.53.253" port: 45889 } attrs { promote: false } } peers { permanent_uuid: "07cfc39c8b0a4037bc133186d8f0c5aa" member_type: NON_VOTER last_known_addr { host: "127.22.53.252" port: 39937 } attrs { promote: true } } }
I20260926 07:30:19.170500 29612 sys_catalog.cc:458] T 00000000000000000000000000000000 P b71a0534077f44f7b2cd8a7fa7688789 [sys.catalog]: This master's current role is: LEADER
I20260926 07:30:19.173609 29587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:19.136961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.195:52293 (local address 127.22.53.253:45889)
0926 07:30:19.137088 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:19.137093 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:19.138059 (+ 966us) server_negotiation.cc:407] Connection header received
0926 07:30:19.140034 (+ 1975us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:19.140038 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:19.140093 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:19.140187 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:19.142024 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:19.142992 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:19.145182 (+ 2190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:19.145546 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:19.145639 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:19.151520 (+ 5881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:19.151526 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:19.151532 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:19.151568 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:19.154424 (+ 2856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:19.159662 (+ 5238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:19.159666 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:19.159668 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:19.159785 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:19.160204 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:19.160210 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:19.160215 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:19.160579 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:19.160677 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:19.173298 (+ 12621us) server_negotiation.cc:299] Negotiation successful
0926 07:30:19.173453 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 07:30:19.174296 30105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:19.137364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.253:45889 (local address 127.22.53.195:52293)
0926 07:30:19.137881 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:19.137940 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:19.139306 (+ 1366us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:19.140231 (+ 925us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:19.140234 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:19.140812 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:19.141816 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:19.141839 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:19.143174 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:19.143178 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:19.145015 (+ 1837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:19.145032 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:19.145855 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:19.145859 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:19.146247 (+ 388us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:19.147362 (+ 1115us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:19.147374 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:19.151305 (+ 3931us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:30:19.159234 (+ 7929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:19.159239 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:19.159243 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:19.159514 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:19.159958 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:19.159961 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:19.159964 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:19.160064 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:19.172847 (+ 12783us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:30:19.172850 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:30:19.173014 (+ 164us) client_negotiation.cc:769] Sending connection context
0926 07:30:19.173838 (+ 824us) client_negotiation.cc:240] Negotiation successful
0926 07:30:19.174070 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"mutex_wait_us":467,"spinlock_wait_cycles":59136,"thread_start_us":129,"threads_started":1}
I20260926 07:30:19.174995 30103 heartbeater.cc:347] Connected to a master server at 127.22.53.253:45889
I20260926 07:30:19.175133 30103 heartbeater.cc:464] Registering TS with master...
I20260926 07:30:19.175487 30103 heartbeater.cc:511] Master 127.22.53.253:45889 requested a full tablet report, sending...
I20260926 07:30:19.176338 29465 ts_manager.cc:194] Registered new tserver with Master: ebec0858532b452191c80c0652eb9b8e (127.22.53.195:42015)
I20260926 07:30:19.176505 29612 raft_consensus.cc:1050] T 00000000000000000000000000000000 P b71a0534077f44f7b2cd8a7fa7688789: attempting to promote NON_VOTER 07cfc39c8b0a4037bc133186d8f0c5aa to VOTER
I20260926 07:30:19.176851 30116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:19.145162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34596 (local address 127.22.53.253:45889)
0926 07:30:19.147807 (+ 2645us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:19.147822 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:19.157458 (+ 9636us) server_negotiation.cc:407] Connection header received
0926 07:30:19.157521 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:19.157524 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:19.157593 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:19.157694 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:19.162860 (+ 5166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:19.163680 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:19.168505 (+ 4825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:19.169004 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:19.169123 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:19.171326 (+ 2203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:19.171331 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:19.171334 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:19.171367 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:19.173415 (+ 2048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:19.174849 (+ 1434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:19.174853 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:19.174856 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:19.174970 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:19.175439 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:19.175442 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:19.175444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:19.175710 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:19.175842 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:19.176512 (+ 670us) server_negotiation.cc:299] Negotiation successful
0926 07:30:19.176690 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2546,"thread_start_us":72,"threads_started":1}
I20260926 07:30:19.176924 29612 consensus_queue.cc:237] T 00000000000000000000000000000000 P b71a0534077f44f7b2cd8a7fa7688789 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b71a0534077f44f7b2cd8a7fa7688789" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 44679 } } peers { permanent_uuid: "5343811b11a34d0483a88e2adef7fc12" member_type: VOTER last_known_addr { host: "127.22.53.253" port: 45889 } attrs { promote: false } } peers { permanent_uuid: "07cfc39c8b0a4037bc133186d8f0c5aa" member_type: VOTER last_known_addr { host: "127.22.53.252" port: 39937 } attrs { promote: false } }
I20260926 07:30:19.178375 29474 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5343811b11a34d0483a88e2adef7fc12 [term 1 FOLLOWER]: Refusing update from remote peer b71a0534077f44f7b2cd8a7fa7688789: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260926 07:30:19.178946 30126 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b71a0534077f44f7b2cd8a7fa7688789 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5343811b11a34d0483a88e2adef7fc12" member_type: VOTER last_known_addr { host: "127.22.53.253" port: 45889 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260926 07:30:19.179160 29658 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 07cfc39c8b0a4037bc133186d8f0c5aa [term 1 LEARNER]: Refusing update from remote peer b71a0534077f44f7b2cd8a7fa7688789: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260926 07:30:19.179557 30126 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b71a0534077f44f7b2cd8a7fa7688789 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07cfc39c8b0a4037bc133186d8f0c5aa" member_type: VOTER last_known_addr { host: "127.22.53.252" port: 39937 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260926 07:30:19.181712 30120 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b71a0534077f44f7b2cd8a7fa7688789 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 07cfc39c8b0a4037bc133186d8f0c5aa (127.22.53.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b71a0534077f44f7b2cd8a7fa7688789" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 44679 } } peers { permanent_uuid: "5343811b11a34d0483a88e2adef7fc12" member_type: VOTER last_known_addr { host: "127.22.53.253" port: 45889 } attrs { promote: false } } peers { permanent_uuid: "07cfc39c8b0a4037bc133186d8f0c5aa" member_type: VOTER last_known_addr { host: "127.22.53.252" port: 39937 } attrs { promote: false } } }
I20260926 07:30:19.183338 30120 sys_catalog.cc:455] T 00000000000000000000000000000000 P b71a0534077f44f7b2cd8a7fa7688789 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "b71a0534077f44f7b2cd8a7fa7688789" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b71a0534077f44f7b2cd8a7fa7688789" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 44679 } } peers { permanent_uuid: "5343811b11a34d0483a88e2adef7fc12" member_type: VOTER last_known_addr { host: "127.22.53.253" port: 45889 } attrs { promote: false } } peers { permanent_uuid: "07cfc39c8b0a4037bc133186d8f0c5aa" member_type: VOTER last_known_addr { host: "127.22.53.252" port: 39937 } attrs { promote: false } } }
I20260926 07:30:19.183470 30120 sys_catalog.cc:458] T 00000000000000000000000000000000 P b71a0534077f44f7b2cd8a7fa7688789 [sys.catalog]: This master's current role is: LEADER
I20260926 07:30:19.183467 30119 sys_catalog.cc:455] T 00000000000000000000000000000000 P b71a0534077f44f7b2cd8a7fa7688789 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b71a0534077f44f7b2cd8a7fa7688789" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b71a0534077f44f7b2cd8a7fa7688789" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 44679 } } peers { permanent_uuid: "5343811b11a34d0483a88e2adef7fc12" member_type: VOTER last_known_addr { host: "127.22.53.253" port: 45889 } attrs { promote: false } } peers { permanent_uuid: "07cfc39c8b0a4037bc133186d8f0c5aa" member_type: VOTER last_known_addr { host: "127.22.53.252" port: 39937 } attrs { promote: false } } }
I20260926 07:30:19.183720 30119 sys_catalog.cc:458] T 00000000000000000000000000000000 P b71a0534077f44f7b2cd8a7fa7688789 [sys.catalog]: This master's current role is: LEADER
I20260926 07:30:19.184036 29658 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 07cfc39c8b0a4037bc133186d8f0c5aa [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 07cfc39c8b0a4037bc133186d8f0c5aa (127.22.53.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b71a0534077f44f7b2cd8a7fa7688789" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 44679 } } peers { permanent_uuid: "5343811b11a34d0483a88e2adef7fc12" member_type: VOTER last_known_addr { host: "127.22.53.253" port: 45889 } attrs { promote: false } } peers { permanent_uuid: "07cfc39c8b0a4037bc133186d8f0c5aa" member_type: VOTER last_known_addr { host: "127.22.53.252" port: 39937 } attrs { promote: false } } }
I20260926 07:30:19.185824 29688 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07cfc39c8b0a4037bc133186d8f0c5aa [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b71a0534077f44f7b2cd8a7fa7688789" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b71a0534077f44f7b2cd8a7fa7688789" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 44679 } } peers { permanent_uuid: "5343811b11a34d0483a88e2adef7fc12" member_type: VOTER last_known_addr { host: "127.22.53.253" port: 45889 } attrs { promote: false } } peers { permanent_uuid: "07cfc39c8b0a4037bc133186d8f0c5aa" member_type: VOTER last_known_addr { host: "127.22.53.252" port: 39937 } attrs { promote: false } } }
I20260926 07:30:19.185968 29688 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07cfc39c8b0a4037bc133186d8f0c5aa [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:30:19.187224 30120 sys_catalog.cc:455] T 00000000000000000000000000000000 P b71a0534077f44f7b2cd8a7fa7688789 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b71a0534077f44f7b2cd8a7fa7688789" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b71a0534077f44f7b2cd8a7fa7688789" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 44679 } } peers { permanent_uuid: "5343811b11a34d0483a88e2adef7fc12" member_type: VOTER last_known_addr { host: "127.22.53.253" port: 45889 } attrs { promote: false } } peers { permanent_uuid: "07cfc39c8b0a4037bc133186d8f0c5aa" member_type: VOTER last_known_addr { host: "127.22.53.252" port: 39937 } attrs { promote: false } } }
I20260926 07:30:19.187363 30120 sys_catalog.cc:458] T 00000000000000000000000000000000 P b71a0534077f44f7b2cd8a7fa7688789 [sys.catalog]: This master's current role is: LEADER
I20260926 07:30:19.188015 29474 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5343811b11a34d0483a88e2adef7fc12 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 07cfc39c8b0a4037bc133186d8f0c5aa (127.22.53.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b71a0534077f44f7b2cd8a7fa7688789" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 44679 } } peers { permanent_uuid: "5343811b11a34d0483a88e2adef7fc12" member_type: VOTER last_known_addr { host: "127.22.53.253" port: 45889 } attrs { promote: false } } peers { permanent_uuid: "07cfc39c8b0a4037bc133186d8f0c5aa" member_type: VOTER last_known_addr { host: "127.22.53.252" port: 39937 } attrs { promote: false } } }
I20260926 07:30:19.192517 30129 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5343811b11a34d0483a88e2adef7fc12 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b71a0534077f44f7b2cd8a7fa7688789" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b71a0534077f44f7b2cd8a7fa7688789" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 44679 } } peers { permanent_uuid: "5343811b11a34d0483a88e2adef7fc12" member_type: VOTER last_known_addr { host: "127.22.53.253" port: 45889 } attrs { promote: false } } peers { permanent_uuid: "07cfc39c8b0a4037bc133186d8f0c5aa" member_type: VOTER last_known_addr { host: "127.22.53.252" port: 39937 } attrs { promote: false } } }
I20260926 07:30:19.192785 30129 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5343811b11a34d0483a88e2adef7fc12 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:30:19.198863 30132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:19.193247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.194:38993 (local address 127.0.0.1:45580)
0926 07:30:19.193517 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:19.193532 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:19.193706 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:19.194633 (+ 927us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:19.194636 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:19.194654 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:30:19.194772 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:19.194782 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:19.195863 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:19.195869 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:19.196900 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:19.196910 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:19.198355 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:19.198362 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:19.198457 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:19.198495 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 07:30:19.198578 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 07:30:19.198675 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":129,"threads_started":1}
I20260926 07:30:19.199007 30131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:19.192820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45580 (local address 127.22.53.194:38993)
0926 07:30:19.193228 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:19.193242 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:19.193622 (+ 380us) server_negotiation.cc:407] Connection header received
0926 07:30:19.193888 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:19.193895 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:19.194166 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:19.194419 (+ 253us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:30:19.194917 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:19.195722 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:19.197044 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:19.197975 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:19.198098 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:19.198281 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:19.198688 (+ 407us) server_negotiation.cc:299] Negotiation successful
0926 07:30:19.198836 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":158,"threads_started":1}
I20260926 07:30:19.199944 30130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:19.191851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.195:42015 (local address 127.0.0.1:38744)
0926 07:30:19.192572 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:19.192593 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:19.192705 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:19.195937 (+ 3232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:19.195941 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:19.195966 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:30:19.196065 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:19.196071 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:19.197459 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:19.197466 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:19.198280 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:19.198289 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:19.199513 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:19.199517 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:19.199614 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:19.199638 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 07:30:19.199717 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 07:30:19.199805 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599}
I20260926 07:30:19.200070 30134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:19.193320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38744 (local address 127.22.53.195:42015)
0926 07:30:19.194617 (+ 1297us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:19.194658 (+ 41us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:19.194686 (+ 28us) server_negotiation.cc:407] Connection header received
0926 07:30:19.195361 (+ 675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:19.195418 (+ 57us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:19.195749 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:19.196000 (+ 251us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:30:19.196213 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:19.197326 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:19.198443 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:19.199355 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:19.199494 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:19.199667 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:19.199764 (+ 97us) server_negotiation.cc:299] Negotiation successful
0926 07:30:19.199895 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1173,"thread_start_us":195,"threads_started":1}
I20260926 07:30:19.201877 29613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:19.192163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.193:35569 (local address 127.0.0.1:53574)
0926 07:30:19.192680 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:19.192695 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:19.192776 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:19.194907 (+ 2131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:19.194912 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:19.194944 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:30:19.195096 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:19.195103 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:19.197131 (+ 2028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:19.197135 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:19.198245 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:19.198256 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:19.201389 (+ 3133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:19.201394 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:19.201497 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:19.201525 (+ 28us) client_negotiation.cc:769] Sending connection context
0926 07:30:19.201626 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 07:30:19.201720 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":141,"threads_started":1}
I20260926 07:30:19.201928 30133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:19.193059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53574 (local address 127.22.53.193:35569)
0926 07:30:19.194104 (+ 1045us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:19.194126 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:19.194164 (+ 38us) server_negotiation.cc:407] Connection header received
0926 07:30:19.194312 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:19.194320 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:19.194629 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:19.195382 (+ 753us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:30:19.195743 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:19.197009 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:19.200286 (+ 3277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:19.201215 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:19.201355 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:19.201520 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:19.201662 (+ 142us) server_negotiation.cc:299] Negotiation successful
0926 07:30:19.201771 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":874,"thread_start_us":313,"threads_started":1}
I20260926 07:30:19.202783 29899 tablet_service.cc:1467] Processing CreateTablet for tablet 13dc78bceb4c470cb8dd0fe9ef398065 (DEFAULT_TABLE table=test-table [id=403dde79e58444eba525f640768755a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:30:19.202890 30035 tablet_service.cc:1467] Processing CreateTablet for tablet 13dc78bceb4c470cb8dd0fe9ef398065 (DEFAULT_TABLE table=test-table [id=403dde79e58444eba525f640768755a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:30:19.203903 30035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13dc78bceb4c470cb8dd0fe9ef398065. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:30:19.203933 29899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13dc78bceb4c470cb8dd0fe9ef398065. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:30:19.208344 29762 tablet_service.cc:1467] Processing CreateTablet for tablet 13dc78bceb4c470cb8dd0fe9ef398065 (DEFAULT_TABLE table=test-table [id=403dde79e58444eba525f640768755a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:30:19.209872 29762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13dc78bceb4c470cb8dd0fe9ef398065. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:30:19.212797 30135 tablet_bootstrap.cc:492] T 13dc78bceb4c470cb8dd0fe9ef398065 P ebec0858532b452191c80c0652eb9b8e: Bootstrap starting.
I20260926 07:30:19.214090 30136 tablet_bootstrap.cc:492] T 13dc78bceb4c470cb8dd0fe9ef398065 P ddbc2223fead4ddfa34d0a137efe01cb: Bootstrap starting.
I20260926 07:30:19.215495 30135 tablet_bootstrap.cc:654] T 13dc78bceb4c470cb8dd0fe9ef398065 P ebec0858532b452191c80c0652eb9b8e: Neither blocks nor log segments found. Creating new log.
I20260926 07:30:19.216408 30136 tablet_bootstrap.cc:654] T 13dc78bceb4c470cb8dd0fe9ef398065 P ddbc2223fead4ddfa34d0a137efe01cb: Neither blocks nor log segments found. Creating new log.
I20260926 07:30:19.216477 30135 log.cc:824] T 13dc78bceb4c470cb8dd0fe9ef398065 P ebec0858532b452191c80c0652eb9b8e: Log is configured to *not* fsync() on all Append() calls
I20260926 07:30:19.217321 30136 log.cc:824] T 13dc78bceb4c470cb8dd0fe9ef398065 P ddbc2223fead4ddfa34d0a137efe01cb: Log is configured to *not* fsync() on all Append() calls
I20260926 07:30:19.218946 30135 tablet_bootstrap.cc:492] T 13dc78bceb4c470cb8dd0fe9ef398065 P ebec0858532b452191c80c0652eb9b8e: No bootstrap required, opened a new log
I20260926 07:30:19.219214 30135 ts_tablet_manager.cc:1397] T 13dc78bceb4c470cb8dd0fe9ef398065 P ebec0858532b452191c80c0652eb9b8e: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260926 07:30:19.219521 30136 tablet_bootstrap.cc:492] T 13dc78bceb4c470cb8dd0fe9ef398065 P ddbc2223fead4ddfa34d0a137efe01cb: No bootstrap required, opened a new log
I20260926 07:30:19.219857 30136 ts_tablet_manager.cc:1397] T 13dc78bceb4c470cb8dd0fe9ef398065 P ddbc2223fead4ddfa34d0a137efe01cb: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 07:30:19.220168 30139 tablet_bootstrap.cc:492] T 13dc78bceb4c470cb8dd0fe9ef398065 P 96b38667cb7946819bcb4dfc480f93be: Bootstrap starting.
I20260926 07:30:19.223017 30139 tablet_bootstrap.cc:654] T 13dc78bceb4c470cb8dd0fe9ef398065 P 96b38667cb7946819bcb4dfc480f93be: Neither blocks nor log segments found. Creating new log.
I20260926 07:30:19.223902 30136 raft_consensus.cc:348] T 13dc78bceb4c470cb8dd0fe9ef398065 P ddbc2223fead4ddfa34d0a137efe01cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebec0858532b452191c80c0652eb9b8e" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 42015 } } peers { permanent_uuid: "96b38667cb7946819bcb4dfc480f93be" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 35569 } } peers { permanent_uuid: "ddbc2223fead4ddfa34d0a137efe01cb" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 38993 } }
I20260926 07:30:19.224095 30139 log.cc:824] T 13dc78bceb4c470cb8dd0fe9ef398065 P 96b38667cb7946819bcb4dfc480f93be: Log is configured to *not* fsync() on all Append() calls
I20260926 07:30:19.224246 30136 raft_consensus.cc:374] T 13dc78bceb4c470cb8dd0fe9ef398065 P ddbc2223fead4ddfa34d0a137efe01cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:30:19.224304 30136 raft_consensus.cc:729] T 13dc78bceb4c470cb8dd0fe9ef398065 P ddbc2223fead4ddfa34d0a137efe01cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddbc2223fead4ddfa34d0a137efe01cb, State: Initialized, Role: FOLLOWER
I20260926 07:30:19.224188 30135 raft_consensus.cc:348] T 13dc78bceb4c470cb8dd0fe9ef398065 P ebec0858532b452191c80c0652eb9b8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebec0858532b452191c80c0652eb9b8e" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 42015 } } peers { permanent_uuid: "96b38667cb7946819bcb4dfc480f93be" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 35569 } } peers { permanent_uuid: "ddbc2223fead4ddfa34d0a137efe01cb" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 38993 } }
I20260926 07:30:19.224501 30135 raft_consensus.cc:374] T 13dc78bceb4c470cb8dd0fe9ef398065 P ebec0858532b452191c80c0652eb9b8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:30:19.224594 30135 raft_consensus.cc:729] T 13dc78bceb4c470cb8dd0fe9ef398065 P ebec0858532b452191c80c0652eb9b8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebec0858532b452191c80c0652eb9b8e, State: Initialized, Role: FOLLOWER
I20260926 07:30:19.224948 30136 consensus_queue.cc:260] T 13dc78bceb4c470cb8dd0fe9ef398065 P ddbc2223fead4ddfa34d0a137efe01cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebec0858532b452191c80c0652eb9b8e" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 42015 } } peers { permanent_uuid: "96b38667cb7946819bcb4dfc480f93be" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 35569 } } peers { permanent_uuid: "ddbc2223fead4ddfa34d0a137efe01cb" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 38993 } }
I20260926 07:30:19.225062 30135 consensus_queue.cc:260] T 13dc78bceb4c470cb8dd0fe9ef398065 P ebec0858532b452191c80c0652eb9b8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebec0858532b452191c80c0652eb9b8e" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 42015 } } peers { permanent_uuid: "96b38667cb7946819bcb4dfc480f93be" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 35569 } } peers { permanent_uuid: "ddbc2223fead4ddfa34d0a137efe01cb" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 38993 } }
I20260926 07:30:19.226106 30135 ts_tablet_manager.cc:1428] T 13dc78bceb4c470cb8dd0fe9ef398065 P ebec0858532b452191c80c0652eb9b8e: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260926 07:30:19.226107 30136 ts_tablet_manager.cc:1428] T 13dc78bceb4c470cb8dd0fe9ef398065 P ddbc2223fead4ddfa34d0a137efe01cb: Time spent starting tablet: real 0.006s user 0.000s sys 0.005s
I20260926 07:30:19.226373 29966 heartbeater.cc:503] Master 127.22.53.254:44679 was elected leader, sending a full tablet report...
I20260926 07:30:19.226473 30102 heartbeater.cc:503] Master 127.22.53.254:44679 was elected leader, sending a full tablet report...
I20260926 07:30:19.226708 30139 tablet_bootstrap.cc:492] T 13dc78bceb4c470cb8dd0fe9ef398065 P 96b38667cb7946819bcb4dfc480f93be: No bootstrap required, opened a new log
I20260926 07:30:19.226917 30139 ts_tablet_manager.cc:1397] T 13dc78bceb4c470cb8dd0fe9ef398065 P 96b38667cb7946819bcb4dfc480f93be: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260926 07:30:19.232810 30139 raft_consensus.cc:348] T 13dc78bceb4c470cb8dd0fe9ef398065 P 96b38667cb7946819bcb4dfc480f93be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebec0858532b452191c80c0652eb9b8e" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 42015 } } peers { permanent_uuid: "96b38667cb7946819bcb4dfc480f93be" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 35569 } } peers { permanent_uuid: "ddbc2223fead4ddfa34d0a137efe01cb" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 38993 } }
I20260926 07:30:19.233210 30139 raft_consensus.cc:374] T 13dc78bceb4c470cb8dd0fe9ef398065 P 96b38667cb7946819bcb4dfc480f93be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:30:19.233284 30139 raft_consensus.cc:729] T 13dc78bceb4c470cb8dd0fe9ef398065 P 96b38667cb7946819bcb4dfc480f93be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96b38667cb7946819bcb4dfc480f93be, State: Initialized, Role: FOLLOWER
I20260926 07:30:19.233861 30139 consensus_queue.cc:260] T 13dc78bceb4c470cb8dd0fe9ef398065 P 96b38667cb7946819bcb4dfc480f93be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebec0858532b452191c80c0652eb9b8e" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 42015 } } peers { permanent_uuid: "96b38667cb7946819bcb4dfc480f93be" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 35569 } } peers { permanent_uuid: "ddbc2223fead4ddfa34d0a137efe01cb" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 38993 } }
I20260926 07:30:19.234815 29829 heartbeater.cc:503] Master 127.22.53.254:44679 was elected leader, sending a full tablet report...
I20260926 07:30:19.235174 30139 ts_tablet_manager.cc:1428] T 13dc78bceb4c470cb8dd0fe9ef398065 P 96b38667cb7946819bcb4dfc480f93be: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260926 07:30:19.246249 30141 raft_consensus.cc:482] T 13dc78bceb4c470cb8dd0fe9ef398065 P ddbc2223fead4ddfa34d0a137efe01cb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:30:19.246465 30141 raft_consensus.cc:504] T 13dc78bceb4c470cb8dd0fe9ef398065 P ddbc2223fead4ddfa34d0a137efe01cb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebec0858532b452191c80c0652eb9b8e" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 42015 } } peers { permanent_uuid: "96b38667cb7946819bcb4dfc480f93be" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 35569 } } peers { permanent_uuid: "ddbc2223fead4ddfa34d0a137efe01cb" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 38993 } }
I20260926 07:30:19.247639 30141 leader_election.cc:288] T 13dc78bceb4c470cb8dd0fe9ef398065 P ddbc2223fead4ddfa34d0a137efe01cb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ebec0858532b452191c80c0652eb9b8e (127.22.53.195:42015), 96b38667cb7946819bcb4dfc480f93be (127.22.53.193:35569)
I20260926 07:30:19.251780 29970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:19.248029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.195:42015 (local address 127.22.53.194:57021)
0926 07:30:19.248179 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:19.248194 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:19.248299 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:19.248635 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:19.248638 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:19.248658 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:30:19.248783 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:19.248798 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:19.249544 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:19.249549 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:19.250389 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:19.250398 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:19.251265 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:19.251269 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:19.251401 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:19.251434 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 07:30:19.251508 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 07:30:19.251598 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260926 07:30:19.251909 30133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:19.248519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.194:58309 (local address 127.22.53.193:35569)
0926 07:30:19.248683 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:19.248687 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:19.248706 (+ 19us) server_negotiation.cc:407] Connection header received
0926 07:30:19.248753 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:19.248755 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:19.248791 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:19.248885 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:30:19.249161 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:19.249701 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:19.250748 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:19.251317 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:19.251430 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:19.251491 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:19.251686 (+ 195us) server_negotiation.cc:299] Negotiation successful
0926 07:30:19.251765 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 07:30:19.251935 29969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:19.248314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.193:35569 (local address 127.22.53.194:58309)
0926 07:30:19.248473 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:19.248486 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:19.248578 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:19.248897 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:19.248900 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:19.248915 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:30:19.249042 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:19.249048 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:19.249824 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:19.249829 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:19.250601 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:19.250610 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:19.251439 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:19.251442 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:19.251548 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:19.251571 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 07:30:19.251664 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 07:30:19.251754 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95}
I20260926 07:30:19.252256 30134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:19.248143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.194:57021 (local address 127.22.53.195:42015)
0926 07:30:19.248300 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:19.248307 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:19.248329 (+ 22us) server_negotiation.cc:407] Connection header received
0926 07:30:19.248398 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:19.248402 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:19.248463 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:19.248595 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:30:19.248910 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:19.249423 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:19.250545 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:19.251128 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:19.251256 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:19.251313 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:19.252025 (+ 712us) server_negotiation.cc:299] Negotiation successful
0926 07:30:19.252126 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 07:30:19.252700 29782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13dc78bceb4c470cb8dd0fe9ef398065" candidate_uuid: "ddbc2223fead4ddfa34d0a137efe01cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96b38667cb7946819bcb4dfc480f93be" is_pre_election: true
I20260926 07:30:19.252789 30055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13dc78bceb4c470cb8dd0fe9ef398065" candidate_uuid: "ddbc2223fead4ddfa34d0a137efe01cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebec0858532b452191c80c0652eb9b8e" is_pre_election: true
I20260926 07:30:19.253067 30055 raft_consensus.cc:2454] T 13dc78bceb4c470cb8dd0fe9ef398065 P ebec0858532b452191c80c0652eb9b8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ddbc2223fead4ddfa34d0a137efe01cb in term 0.
I20260926 07:30:19.253067 29782 raft_consensus.cc:2454] T 13dc78bceb4c470cb8dd0fe9ef398065 P 96b38667cb7946819bcb4dfc480f93be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ddbc2223fead4ddfa34d0a137efe01cb in term 0.
I20260926 07:30:19.253625 29852 leader_election.cc:302] T 13dc78bceb4c470cb8dd0fe9ef398065 P ddbc2223fead4ddfa34d0a137efe01cb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ddbc2223fead4ddfa34d0a137efe01cb, ebec0858532b452191c80c0652eb9b8e; no voters:
I20260926 07:30:19.253924 30141 raft_consensus.cc:2781] T 13dc78bceb4c470cb8dd0fe9ef398065 P ddbc2223fead4ddfa34d0a137efe01cb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:30:19.254033 30141 raft_consensus.cc:482] T 13dc78bceb4c470cb8dd0fe9ef398065 P ddbc2223fead4ddfa34d0a137efe01cb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:30:19.254084 30141 raft_consensus.cc:3037] T 13dc78bceb4c470cb8dd0fe9ef398065 P ddbc2223fead4ddfa34d0a137efe01cb [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:30:19.255229 30141 raft_consensus.cc:504] T 13dc78bceb4c470cb8dd0fe9ef398065 P ddbc2223fead4ddfa34d0a137efe01cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebec0858532b452191c80c0652eb9b8e" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 42015 } } peers { permanent_uuid: "96b38667cb7946819bcb4dfc480f93be" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 35569 } } peers { permanent_uuid: "ddbc2223fead4ddfa34d0a137efe01cb" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 38993 } }
I20260926 07:30:19.255637 30141 leader_election.cc:288] T 13dc78bceb4c470cb8dd0fe9ef398065 P ddbc2223fead4ddfa34d0a137efe01cb [CANDIDATE]: Term 1 election: Requested vote from peers ebec0858532b452191c80c0652eb9b8e (127.22.53.195:42015), 96b38667cb7946819bcb4dfc480f93be (127.22.53.193:35569)
I20260926 07:30:19.256227 29782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13dc78bceb4c470cb8dd0fe9ef398065" candidate_uuid: "ddbc2223fead4ddfa34d0a137efe01cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96b38667cb7946819bcb4dfc480f93be"
I20260926 07:30:19.256227 30055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13dc78bceb4c470cb8dd0fe9ef398065" candidate_uuid: "ddbc2223fead4ddfa34d0a137efe01cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebec0858532b452191c80c0652eb9b8e"
I20260926 07:30:19.256399 30055 raft_consensus.cc:3037] T 13dc78bceb4c470cb8dd0fe9ef398065 P ebec0858532b452191c80c0652eb9b8e [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:30:19.256366 29782 raft_consensus.cc:3037] T 13dc78bceb4c470cb8dd0fe9ef398065 P 96b38667cb7946819bcb4dfc480f93be [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:30:19.257586 30055 raft_consensus.cc:2454] T 13dc78bceb4c470cb8dd0fe9ef398065 P ebec0858532b452191c80c0652eb9b8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddbc2223fead4ddfa34d0a137efe01cb in term 1.
I20260926 07:30:19.257597 29782 raft_consensus.cc:2454] T 13dc78bceb4c470cb8dd0fe9ef398065 P 96b38667cb7946819bcb4dfc480f93be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddbc2223fead4ddfa34d0a137efe01cb in term 1.
I20260926 07:30:19.257977 29852 leader_election.cc:302] T 13dc78bceb4c470cb8dd0fe9ef398065 P ddbc2223fead4ddfa34d0a137efe01cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96b38667cb7946819bcb4dfc480f93be, ddbc2223fead4ddfa34d0a137efe01cb; no voters:
I20260926 07:30:19.258177 30141 raft_consensus.cc:2781] T 13dc78bceb4c470cb8dd0fe9ef398065 P ddbc2223fead4ddfa34d0a137efe01cb [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:30:19.258440 30141 raft_consensus.cc:686] T 13dc78bceb4c470cb8dd0fe9ef398065 P ddbc2223fead4ddfa34d0a137efe01cb [term 1 LEADER]: Becoming Leader. State: Replica: ddbc2223fead4ddfa34d0a137efe01cb, State: Running, Role: LEADER
I20260926 07:30:19.258876 30141 consensus_queue.cc:237] T 13dc78bceb4c470cb8dd0fe9ef398065 P ddbc2223fead4ddfa34d0a137efe01cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebec0858532b452191c80c0652eb9b8e" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 42015 } } peers { permanent_uuid: "96b38667cb7946819bcb4dfc480f93be" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 35569 } } peers { permanent_uuid: "ddbc2223fead4ddfa34d0a137efe01cb" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 38993 } }
I20260926 07:30:19.262733 28864 catalog_manager.cc:5495] T 13dc78bceb4c470cb8dd0fe9ef398065 P ddbc2223fead4ddfa34d0a137efe01cb reported cstate change: term changed from 0 to 1, leader changed from <none> to ddbc2223fead4ddfa34d0a137efe01cb (127.22.53.194). New cstate: current_term: 1 leader_uuid: "ddbc2223fead4ddfa34d0a137efe01cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebec0858532b452191c80c0652eb9b8e" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 42015 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96b38667cb7946819bcb4dfc480f93be" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 35569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ddbc2223fead4ddfa34d0a137efe01cb" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 38993 } health_report { overall_health: HEALTHY } } }
I20260926 07:30:19.298254 30131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:19.294335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45588 (local address 127.22.53.194:38993)
0926 07:30:19.294479 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:19.294483 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:19.294499 (+ 16us) server_negotiation.cc:407] Connection header received
0926 07:30:19.294698 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:19.294701 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:19.294762 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:19.294859 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:30:19.295221 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:19.295746 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:19.296758 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:19.296997 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:19.297091 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:19.297374 (+ 283us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:30:19.297843 (+ 469us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:30:19.297938 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:19.298038 (+ 100us) server_negotiation.cc:299] Negotiation successful
0926 07:30:19.298115 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 07:30:19.305461 30055 raft_consensus.cc:1261] T 13dc78bceb4c470cb8dd0fe9ef398065 P ebec0858532b452191c80c0652eb9b8e [term 1 FOLLOWER]: Refusing update from remote peer ddbc2223fead4ddfa34d0a137efe01cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:30:19.305526 29782 raft_consensus.cc:1261] T 13dc78bceb4c470cb8dd0fe9ef398065 P 96b38667cb7946819bcb4dfc480f93be [term 1 FOLLOWER]: Refusing update from remote peer ddbc2223fead4ddfa34d0a137efe01cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:30:19.306416 30141 consensus_queue.cc:1035] T 13dc78bceb4c470cb8dd0fe9ef398065 P ddbc2223fead4ddfa34d0a137efe01cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "96b38667cb7946819bcb4dfc480f93be" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 35569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:30:19.306672 30144 consensus_queue.cc:1035] T 13dc78bceb4c470cb8dd0fe9ef398065 P ddbc2223fead4ddfa34d0a137efe01cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebec0858532b452191c80c0652eb9b8e" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 42015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:30:19.320303 30149 mvcc.cc:205] Tried to move back new op lower bound from 7333510427865956352 to 7333510427687432192. Current Snapshot: MvccSnapshot[applied={T|T < 7333510427865956352}]
I20260926 07:30:19.331248 28864 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44742:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:30:19.331489 28864 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44742:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:30:19.337312 28864 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b71a0534077f44f7b2cd8a7fa7688789: Sending DeleteTablet for 3 replicas of tablet 13dc78bceb4c470cb8dd0fe9ef398065
I20260926 07:30:19.338119 30035 tablet_service.cc:1514] Processing DeleteTablet for tablet 13dc78bceb4c470cb8dd0fe9ef398065 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:30:19 UTC) from {username='oryx', principal='oryx/127.22.53.254@KRBTEST.COM'} at 127.0.0.1:38744
I20260926 07:30:19.338227 29899 tablet_service.cc:1514] Processing DeleteTablet for tablet 13dc78bceb4c470cb8dd0fe9ef398065 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:30:19 UTC) from {username='oryx', principal='oryx/127.22.53.254@KRBTEST.COM'} at 127.0.0.1:45580
I20260926 07:30:19.338788 30157 tablet_replica.cc:330] T 13dc78bceb4c470cb8dd0fe9ef398065 P ebec0858532b452191c80c0652eb9b8e: stopping tablet replica
I20260926 07:30:19.338789 29762 tablet_service.cc:1514] Processing DeleteTablet for tablet 13dc78bceb4c470cb8dd0fe9ef398065 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:30:19 UTC) from {username='oryx', principal='oryx/127.22.53.254@KRBTEST.COM'} at 127.0.0.1:53574
I20260926 07:30:19.339303 30159 tablet_replica.cc:330] T 13dc78bceb4c470cb8dd0fe9ef398065 P 96b38667cb7946819bcb4dfc480f93be: stopping tablet replica
I20260926 07:30:19.339422 30157 raft_consensus.cc:2229] T 13dc78bceb4c470cb8dd0fe9ef398065 P ebec0858532b452191c80c0652eb9b8e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:30:19.339598 30159 raft_consensus.cc:2229] T 13dc78bceb4c470cb8dd0fe9ef398065 P 96b38667cb7946819bcb4dfc480f93be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:30:19.339946 30159 raft_consensus.cc:2258] T 13dc78bceb4c470cb8dd0fe9ef398065 P 96b38667cb7946819bcb4dfc480f93be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:30:19.339986 30157 raft_consensus.cc:2258] T 13dc78bceb4c470cb8dd0fe9ef398065 P ebec0858532b452191c80c0652eb9b8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:30:19.341818 30157 ts_tablet_manager.cc:1905] T 13dc78bceb4c470cb8dd0fe9ef398065 P ebec0858532b452191c80c0652eb9b8e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:30:19.341818 30159 ts_tablet_manager.cc:1905] T 13dc78bceb4c470cb8dd0fe9ef398065 P 96b38667cb7946819bcb4dfc480f93be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:30:19.342912 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 28832
I20260926 07:30:19.345407 30159 ts_tablet_manager.cc:1918] T 13dc78bceb4c470cb8dd0fe9ef398065 P 96b38667cb7946819bcb4dfc480f93be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:30:19.345546 30159 log.cc:1196] T 13dc78bceb4c470cb8dd0fe9ef398065 P 96b38667cb7946819bcb4dfc480f93be: Deleting WAL directory at /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/ts-0/wal/wals/13dc78bceb4c470cb8dd0fe9ef398065
I20260926 07:30:19.345948 30159 ts_tablet_manager.cc:1939] T 13dc78bceb4c470cb8dd0fe9ef398065 P 96b38667cb7946819bcb4dfc480f93be: Deleting consensus metadata
I20260926 07:30:19.346112 30157 ts_tablet_manager.cc:1918] T 13dc78bceb4c470cb8dd0fe9ef398065 P ebec0858532b452191c80c0652eb9b8e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:30:19.346280 30157 log.cc:1196] T 13dc78bceb4c470cb8dd0fe9ef398065 P ebec0858532b452191c80c0652eb9b8e: Deleting WAL directory at /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/ts-2/wal/wals/13dc78bceb4c470cb8dd0fe9ef398065
I20260926 07:30:19.346686 30157 ts_tablet_manager.cc:1939] T 13dc78bceb4c470cb8dd0fe9ef398065 P ebec0858532b452191c80c0652eb9b8e: Deleting consensus metadata
I20260926 07:30:19.348801 30158 tablet_replica.cc:330] T 13dc78bceb4c470cb8dd0fe9ef398065 P ddbc2223fead4ddfa34d0a137efe01cb: stopping tablet replica
I20260926 07:30:19.349165 30158 raft_consensus.cc:2229] T 13dc78bceb4c470cb8dd0fe9ef398065 P ddbc2223fead4ddfa34d0a137efe01cb [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:30:19.349582 30158 raft_consensus.cc:2258] T 13dc78bceb4c470cb8dd0fe9ef398065 P ddbc2223fead4ddfa34d0a137efe01cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:30:19.351028 30158 ts_tablet_manager.cc:1905] T 13dc78bceb4c470cb8dd0fe9ef398065 P ddbc2223fead4ddfa34d0a137efe01cb: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 07:30:19.351449 29718 connection.cc:535] server connection from 127.0.0.1:53574 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260926 07:30:19.351482 29990 connection.cc:535] server connection from 127.0.0.1:38744 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 07:30:19.355406 30158 ts_tablet_manager.cc:1918] T 13dc78bceb4c470cb8dd0fe9ef398065 P ddbc2223fead4ddfa34d0a137efe01cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:30:19.355561 30158 log.cc:1196] T 13dc78bceb4c470cb8dd0fe9ef398065 P ddbc2223fead4ddfa34d0a137efe01cb: Deleting WAL directory at /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/ts-1/wal/wals/13dc78bceb4c470cb8dd0fe9ef398065
I20260926 07:30:19.355952 30158 ts_tablet_manager.cc:1939] T 13dc78bceb4c470cb8dd0fe9ef398065 P ddbc2223fead4ddfa34d0a137efe01cb: Deleting consensus metadata
W20260926 07:30:19.358385 29854 connection.cc:441] server connection from 127.0.0.1:45580 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:45580 (request call id 1) could send its response
I20260926 07:30:19.367362 30116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:19.342421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34604 (local address 127.22.53.253:45889)
0926 07:30:19.342537 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:19.342553 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:19.356829 (+ 14276us) server_negotiation.cc:407] Connection header received
0926 07:30:19.356999 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:19.357002 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:19.357065 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:19.357134 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:19.358068 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:19.358830 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:19.360174 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:19.360671 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:19.360766 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:19.362925 (+ 2159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:19.362929 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:19.362931 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:19.362960 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:19.365153 (+ 2193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:19.365687 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:19.365693 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:19.365698 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:19.365791 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:19.366089 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:19.366095 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:19.366100 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:19.366365 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:19.366494 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:19.367018 (+ 524us) server_negotiation.cc:299] Negotiation successful
0926 07:30:19.367162 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":46,"server-negotiator.queue_time_us":36}
I20260926 07:30:19.367367 29587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:19.355811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34610 (local address 127.22.53.253:45889)
0926 07:30:19.355941 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:19.355945 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:19.356006 (+ 61us) server_negotiation.cc:407] Connection header received
0926 07:30:19.356315 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:19.356319 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:19.356380 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:19.356473 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:19.357470 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:19.358250 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:19.360133 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:19.360616 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:19.360691 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:19.362878 (+ 2187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:19.362882 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:19.362885 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:19.362922 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:19.365639 (+ 2717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:19.366115 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:19.366118 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:19.366119 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:19.366214 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:19.366526 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:19.366530 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:19.366531 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:19.366717 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:19.366813 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:19.367162 (+ 349us) server_negotiation.cc:299] Negotiation successful
0926 07:30:19.367258 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":6,"server-negotiator.queue_time_us":51,"spinlock_wait_cycles":896}
I20260926 07:30:19.370034 29834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:19.358672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56464 (local address 127.22.53.252:39937)
0926 07:30:19.358821 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:19.358827 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:19.358839 (+ 12us) server_negotiation.cc:407] Connection header received
0926 07:30:19.358882 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:19.358884 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:19.358932 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:19.359005 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:19.360557 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:19.361299 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:19.362498 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:19.362870 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:19.362946 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:19.364860 (+ 1914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:19.364865 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:19.364868 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:19.364907 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:19.366705 (+ 1798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:19.367276 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:19.367280 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:19.367282 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:19.367370 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:19.367674 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:19.367677 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:19.367679 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:19.367892 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:19.367999 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:19.369710 (+ 1711us) server_negotiation.cc:299] Negotiation successful
0926 07:30:19.369843 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 07:30:19.377589 30113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:19.342075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56448 (local address 127.22.53.252:39937)
0926 07:30:19.342202 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:19.342207 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:19.368162 (+ 25955us) server_negotiation.cc:407] Connection header received
0926 07:30:19.368205 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:19.368208 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:19.368261 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:19.368331 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:19.369394 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:19.370037 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:19.371411 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:19.371654 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:19.371909 (+ 255us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:19.373654 (+ 1745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:19.373657 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:19.373660 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:19.373683 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:19.375432 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:19.375990 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:19.375993 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:19.375995 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:19.376089 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:19.376405 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:19.376408 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:19.376410 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:19.376710 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:19.376857 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:19.377229 (+ 372us) server_negotiation.cc:299] Negotiation successful
0926 07:30:19.377389 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 07:30:20.321192 30167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:20.320590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:44679 (local address 127.22.53.195:35601)
0926 07:30:20.320916 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:20.321024 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:44679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":83,"threads_started":1}
W20260926 07:30:20.321511 30102 heartbeater.cc:644] Failed to heartbeat to 127.22.53.254:44679 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.22.53.254:44679: connect: Connection refused (error 111)
I20260926 07:30:20.322765 30168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:20.322287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:44679 (local address 127.22.53.193:48563)
0926 07:30:20.322562 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:20.322650 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:44679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":99,"threads_started":1}
W20260926 07:30:20.323096 29829 heartbeater.cc:644] Failed to heartbeat to 127.22.53.254:44679 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.22.53.254:44679: connect: Connection refused (error 111)
I20260926 07:30:20.325948 30169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:20.325443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:44679 (local address 127.22.53.194:49881)
0926 07:30:20.325734 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:20.325832 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:44679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":76,"threads_started":1}
W20260926 07:30:20.326259 29966 heartbeater.cc:644] Failed to heartbeat to 127.22.53.254:44679 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.22.53.254:44679: connect: Connection refused (error 111)
I20260926 07:30:20.837167 30170 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5343811b11a34d0483a88e2adef7fc12 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b71a0534077f44f7b2cd8a7fa7688789)
I20260926 07:30:20.837251 30171 raft_consensus.cc:482] T 00000000000000000000000000000000 P 07cfc39c8b0a4037bc133186d8f0c5aa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b71a0534077f44f7b2cd8a7fa7688789)
I20260926 07:30:20.837427 30170 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5343811b11a34d0483a88e2adef7fc12 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b71a0534077f44f7b2cd8a7fa7688789" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 44679 } } peers { permanent_uuid: "5343811b11a34d0483a88e2adef7fc12" member_type: VOTER last_known_addr { host: "127.22.53.253" port: 45889 } attrs { promote: false } } peers { permanent_uuid: "07cfc39c8b0a4037bc133186d8f0c5aa" member_type: VOTER last_known_addr { host: "127.22.53.252" port: 39937 } attrs { promote: false } }
I20260926 07:30:20.837461 30171 raft_consensus.cc:504] T 00000000000000000000000000000000 P 07cfc39c8b0a4037bc133186d8f0c5aa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b71a0534077f44f7b2cd8a7fa7688789" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 44679 } } peers { permanent_uuid: "5343811b11a34d0483a88e2adef7fc12" member_type: VOTER last_known_addr { host: "127.22.53.253" port: 45889 } attrs { promote: false } } peers { permanent_uuid: "07cfc39c8b0a4037bc133186d8f0c5aa" member_type: VOTER last_known_addr { host: "127.22.53.252" port: 39937 } attrs { promote: false } }
I20260926 07:30:20.838616 30170 leader_election.cc:288] T 00000000000000000000000000000000 P 5343811b11a34d0483a88e2adef7fc12 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b71a0534077f44f7b2cd8a7fa7688789 (127.22.53.254:44679), 07cfc39c8b0a4037bc133186d8f0c5aa (127.22.53.252:39937)
I20260926 07:30:20.838634 30171 leader_election.cc:288] T 00000000000000000000000000000000 P 07cfc39c8b0a4037bc133186d8f0c5aa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b71a0534077f44f7b2cd8a7fa7688789 (127.22.53.254:44679), 5343811b11a34d0483a88e2adef7fc12 (127.22.53.253:45889)
I20260926 07:30:20.839303 30172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:20.838717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:44679 (local address 127.0.0.1:45124)
0926 07:30:20.839004 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:20.839155 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:44679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":86,"threads_started":1}
W20260926 07:30:20.839581 29633 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.53.254:44679: connect: Connection refused (error 111)
I20260926 07:30:20.840304 30176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:20.839283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:44679 (local address 127.0.0.1:45134)
0926 07:30:20.839847 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:20.839923 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:44679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":65,"threads_started":1}
W20260926 07:30:20.840482 29449 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.53.254:44679: connect: Connection refused (error 111)
I20260926 07:30:20.840937 30174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:20.840360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:44679 (local address 127.0.0.1:45136)
0926 07:30:20.840801 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:20.840863 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:44679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348}
W20260926 07:30:20.841131 29633 leader_election.cc:334] T 00000000000000000000000000000000 P 07cfc39c8b0a4037bc133186d8f0c5aa [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b71a0534077f44f7b2cd8a7fa7688789 (127.22.53.254:44679): Network error: Client connection negotiation failed: client connection to 127.22.53.254:44679: connect: Connection refused (error 111)
I20260926 07:30:20.841500 30176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:20.841239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:44679 (local address 127.0.0.1:45138)
0926 07:30:20.841371 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:20.841427 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:44679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260926 07:30:20.841730 29449 leader_election.cc:334] T 00000000000000000000000000000000 P 5343811b11a34d0483a88e2adef7fc12 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b71a0534077f44f7b2cd8a7fa7688789 (127.22.53.254:44679): Network error: Client connection negotiation failed: client connection to 127.22.53.254:44679: connect: Connection refused (error 111)
I20260926 07:30:20.843979 30175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:20.839279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56470 (local address 127.22.53.252:39937)
0926 07:30:20.839489 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:20.839493 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:20.839711 (+ 218us) server_negotiation.cc:407] Connection header received
0926 07:30:20.839920 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:20.839926 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:20.840066 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:20.840167 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:30:20.840725 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:20.841706 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:20.842989 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:20.843398 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:20.843528 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:20.843593 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:20.843769 (+ 176us) server_negotiation.cc:299] Negotiation successful
0926 07:30:20.843850 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":53,"threads_started":1}
I20260926 07:30:20.843994 30173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:20.838955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.252:39937 (local address 127.0.0.1:56470)
0926 07:30:20.839633 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:20.839646 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:20.839787 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:20.840380 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:20.840385 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:20.840410 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:30:20.840572 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:20.840583 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:20.841833 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:20.841838 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:20.842864 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:20.842874 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:20.843532 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:20.843535 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:20.843628 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:20.843652 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 07:30:20.843754 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 07:30:20.843837 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":611,"thread_start_us":70,"threads_started":1}
I20260926 07:30:20.844426 30177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:20.839320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34624 (local address 127.22.53.253:45889)
0926 07:30:20.839731 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:20.839735 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:20.840474 (+ 739us) server_negotiation.cc:407] Connection header received
0926 07:30:20.840711 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:20.840714 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:20.840760 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:20.840834 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:30:20.841312 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:20.842128 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:20.843213 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:20.843624 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:20.843745 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:20.843803 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:20.843972 (+ 169us) server_negotiation.cc:299] Negotiation successful
0926 07:30:20.844042 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":67,"threads_started":1}
I20260926 07:30:20.844444 30172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:20.839045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.253:45889 (local address 127.0.0.1:34624)
0926 07:30:20.840213 (+ 1168us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:20.840227 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:20.840516 (+ 289us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:20.840919 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:20.840922 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:20.840939 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:30:20.841155 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:20.841167 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:20.842237 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:20.842241 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:20.843088 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:20.843097 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:20.843748 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:20.843752 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:20.843850 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:20.843869 (+ 19us) client_negotiation.cc:769] Sending connection context
0926 07:30:20.843957 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 07:30:20.844036 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1048,"mutex_wait_us":2,"thread_start_us":72,"threads_started":1}
I20260926 07:30:20.844764 29658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5343811b11a34d0483a88e2adef7fc12" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "07cfc39c8b0a4037bc133186d8f0c5aa" is_pre_election: true
I20260926 07:30:20.844911 29474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "07cfc39c8b0a4037bc133186d8f0c5aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "5343811b11a34d0483a88e2adef7fc12" is_pre_election: true
I20260926 07:30:20.845006 29658 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 07cfc39c8b0a4037bc133186d8f0c5aa [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5343811b11a34d0483a88e2adef7fc12 in term 1.
I20260926 07:30:20.845148 29474 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5343811b11a34d0483a88e2adef7fc12 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07cfc39c8b0a4037bc133186d8f0c5aa in term 1.
I20260926 07:30:20.845439 29450 leader_election.cc:302] T 00000000000000000000000000000000 P 5343811b11a34d0483a88e2adef7fc12 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 07cfc39c8b0a4037bc133186d8f0c5aa, 5343811b11a34d0483a88e2adef7fc12; no voters: b71a0534077f44f7b2cd8a7fa7688789
I20260926 07:30:20.845649 30170 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5343811b11a34d0483a88e2adef7fc12 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260926 07:30:20.845706 30170 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5343811b11a34d0483a88e2adef7fc12 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b71a0534077f44f7b2cd8a7fa7688789)
I20260926 07:30:20.845745 30170 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5343811b11a34d0483a88e2adef7fc12 [term 1 FOLLOWER]: Advancing to term 2
I20260926 07:30:20.845963 29633 leader_election.cc:302] T 00000000000000000000000000000000 P 07cfc39c8b0a4037bc133186d8f0c5aa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 07cfc39c8b0a4037bc133186d8f0c5aa, 5343811b11a34d0483a88e2adef7fc12; no voters: b71a0534077f44f7b2cd8a7fa7688789
I20260926 07:30:20.846203 30171 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 07cfc39c8b0a4037bc133186d8f0c5aa [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260926 07:30:20.846263 30171 raft_consensus.cc:482] T 00000000000000000000000000000000 P 07cfc39c8b0a4037bc133186d8f0c5aa [term 1 FOLLOWER]: Starting leader election (detected failure of leader b71a0534077f44f7b2cd8a7fa7688789)
I20260926 07:30:20.846303 30171 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 07cfc39c8b0a4037bc133186d8f0c5aa [term 1 FOLLOWER]: Advancing to term 2
I20260926 07:30:20.846969 30170 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5343811b11a34d0483a88e2adef7fc12 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b71a0534077f44f7b2cd8a7fa7688789" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 44679 } } peers { permanent_uuid: "5343811b11a34d0483a88e2adef7fc12" member_type: VOTER last_known_addr { host: "127.22.53.253" port: 45889 } attrs { promote: false } } peers { permanent_uuid: "07cfc39c8b0a4037bc133186d8f0c5aa" member_type: VOTER last_known_addr { host: "127.22.53.252" port: 39937 } attrs { promote: false } }
I20260926 07:30:20.847391 30170 leader_election.cc:288] T 00000000000000000000000000000000 P 5343811b11a34d0483a88e2adef7fc12 [CANDIDATE]: Term 2 election: Requested vote from peers b71a0534077f44f7b2cd8a7fa7688789 (127.22.53.254:44679), 07cfc39c8b0a4037bc133186d8f0c5aa (127.22.53.252:39937)
I20260926 07:30:20.847352 30171 raft_consensus.cc:504] T 00000000000000000000000000000000 P 07cfc39c8b0a4037bc133186d8f0c5aa [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b71a0534077f44f7b2cd8a7fa7688789" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 44679 } } peers { permanent_uuid: "5343811b11a34d0483a88e2adef7fc12" member_type: VOTER last_known_addr { host: "127.22.53.253" port: 45889 } attrs { promote: false } } peers { permanent_uuid: "07cfc39c8b0a4037bc133186d8f0c5aa" member_type: VOTER last_known_addr { host: "127.22.53.252" port: 39937 } attrs { promote: false } }
I20260926 07:30:20.847738 30171 leader_election.cc:288] T 00000000000000000000000000000000 P 07cfc39c8b0a4037bc133186d8f0c5aa [CANDIDATE]: Term 2 election: Requested vote from peers b71a0534077f44f7b2cd8a7fa7688789 (127.22.53.254:44679), 5343811b11a34d0483a88e2adef7fc12 (127.22.53.253:45889)
I20260926 07:30:20.847764 29658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5343811b11a34d0483a88e2adef7fc12" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "07cfc39c8b0a4037bc133186d8f0c5aa"
I20260926 07:30:20.847952 29658 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 07cfc39c8b0a4037bc133186d8f0c5aa [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 5343811b11a34d0483a88e2adef7fc12 in current term 2: Already voted for candidate 07cfc39c8b0a4037bc133186d8f0c5aa in this term.
I20260926 07:30:20.848018 30173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:20.847541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:44679 (local address 127.0.0.1:45144)
0926 07:30:20.847777 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:20.847838 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:44679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152}
I20260926 07:30:20.848157 30172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:20.847835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:44679 (local address 127.0.0.1:45148)
0926 07:30:20.848031 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:20.848096 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:44679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260926 07:30:20.848349 29474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "07cfc39c8b0a4037bc133186d8f0c5aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "5343811b11a34d0483a88e2adef7fc12"
I20260926 07:30:20.848541 29474 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 5343811b11a34d0483a88e2adef7fc12 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 07cfc39c8b0a4037bc133186d8f0c5aa in current term 2: Already voted for candidate 5343811b11a34d0483a88e2adef7fc12 in this term.
I20260926 07:30:20.848757 30173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:20.848502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:44679 (local address 127.0.0.1:45164)
0926 07:30:20.848616 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:20.848662 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:44679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260926 07:30:20.848846 29449 leader_election.cc:334] T 00000000000000000000000000000000 P 5343811b11a34d0483a88e2adef7fc12 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer b71a0534077f44f7b2cd8a7fa7688789 (127.22.53.254:44679): Network error: Client connection negotiation failed: client connection to 127.22.53.254:44679: connect: Connection refused (error 111)
I20260926 07:30:20.848930 29449 leader_election.cc:302] T 00000000000000000000000000000000 P 5343811b11a34d0483a88e2adef7fc12 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5343811b11a34d0483a88e2adef7fc12; no voters: 07cfc39c8b0a4037bc133186d8f0c5aa, b71a0534077f44f7b2cd8a7fa7688789
I20260926 07:30:20.849216 30170 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 5343811b11a34d0483a88e2adef7fc12 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260926 07:30:20.849311 30172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:20.849069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:44679 (local address 127.0.0.1:45174)
0926 07:30:20.849181 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:20.849240 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:44679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260926 07:30:20.849407 29633 leader_election.cc:334] T 00000000000000000000000000000000 P 07cfc39c8b0a4037bc133186d8f0c5aa [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer b71a0534077f44f7b2cd8a7fa7688789 (127.22.53.254:44679): Network error: Client connection negotiation failed: client connection to 127.22.53.254:44679: connect: Connection refused (error 111)
I20260926 07:30:20.849486 29633 leader_election.cc:302] T 00000000000000000000000000000000 P 07cfc39c8b0a4037bc133186d8f0c5aa [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 07cfc39c8b0a4037bc133186d8f0c5aa; no voters: 5343811b11a34d0483a88e2adef7fc12, b71a0534077f44f7b2cd8a7fa7688789
I20260926 07:30:20.849651 30171 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 07cfc39c8b0a4037bc133186d8f0c5aa [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260926 07:30:21.322947 30180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:21.322210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:44679 (local address 127.22.53.195:36393)
0926 07:30:21.322486 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:21.322589 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:44679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":78,"threads_started":1}
I20260926 07:30:21.323997 30181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:21.323610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:44679 (local address 127.22.53.193:57887)
0926 07:30:21.323861 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:21.323917 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:44679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":83,"threads_started":1}
I20260926 07:30:21.327346 30182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:21.326818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:44679 (local address 127.22.53.194:38061)
0926 07:30:21.327160 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:21.327243 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:44679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":80,"threads_started":1}
I20260926 07:30:22.325280 30183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:22.324152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:44679 (local address 127.22.53.195:56145)
0926 07:30:22.325062 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:22.325148 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:44679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":816,"thread_start_us":105,"threads_started":1}
I20260926 07:30:22.325678 30184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:22.324876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:44679 (local address 127.22.53.193:44217)
0926 07:30:22.325481 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:22.325547 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:44679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":72,"threads_started":1}
I20260926 07:30:22.328508 30185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:22.328074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:44679 (local address 127.22.53.194:56547)
0926 07:30:22.328336 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:22.328409 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:44679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":98,"threads_started":1}
I20260926 07:30:22.635082 30186 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5343811b11a34d0483a88e2adef7fc12 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:30:22.635228 30186 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5343811b11a34d0483a88e2adef7fc12 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b71a0534077f44f7b2cd8a7fa7688789" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 44679 } } peers { permanent_uuid: "5343811b11a34d0483a88e2adef7fc12" member_type: VOTER last_known_addr { host: "127.22.53.253" port: 45889 } attrs { promote: false } } peers { permanent_uuid: "07cfc39c8b0a4037bc133186d8f0c5aa" member_type: VOTER last_known_addr { host: "127.22.53.252" port: 39937 } attrs { promote: false } }
I20260926 07:30:22.635635 30186 leader_election.cc:288] T 00000000000000000000000000000000 P 5343811b11a34d0483a88e2adef7fc12 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b71a0534077f44f7b2cd8a7fa7688789 (127.22.53.254:44679), 07cfc39c8b0a4037bc133186d8f0c5aa (127.22.53.252:39937)
I20260926 07:30:22.636121 29658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5343811b11a34d0483a88e2adef7fc12" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "07cfc39c8b0a4037bc133186d8f0c5aa" is_pre_election: true
I20260926 07:30:22.636248 30187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:22.635870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:44679 (local address 127.0.0.1:45454)
0926 07:30:22.636098 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:22.636162 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:44679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":80,"threads_started":1}
I20260926 07:30:22.636301 29658 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 07cfc39c8b0a4037bc133186d8f0c5aa [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5343811b11a34d0483a88e2adef7fc12 in term 2.
I20260926 07:30:22.636703 29450 leader_election.cc:302] T 00000000000000000000000000000000 P 5343811b11a34d0483a88e2adef7fc12 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07cfc39c8b0a4037bc133186d8f0c5aa, 5343811b11a34d0483a88e2adef7fc12; no voters:
I20260926 07:30:22.636942 30186 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5343811b11a34d0483a88e2adef7fc12 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260926 07:30:22.637022 30186 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5343811b11a34d0483a88e2adef7fc12 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:30:22.637066 30186 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5343811b11a34d0483a88e2adef7fc12 [term 2 FOLLOWER]: Advancing to term 3
I20260926 07:30:22.637291 30187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:22.636985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:44679 (local address 127.0.0.1:45458)
0926 07:30:22.637163 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:22.637227 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:44679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
W20260926 07:30:22.637475 29449 leader_election.cc:334] T 00000000000000000000000000000000 P 5343811b11a34d0483a88e2adef7fc12 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b71a0534077f44f7b2cd8a7fa7688789 (127.22.53.254:44679): Network error: Client connection negotiation failed: client connection to 127.22.53.254:44679: connect: Connection refused (error 111)
I20260926 07:30:22.638262 30186 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5343811b11a34d0483a88e2adef7fc12 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b71a0534077f44f7b2cd8a7fa7688789" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 44679 } } peers { permanent_uuid: "5343811b11a34d0483a88e2adef7fc12" member_type: VOTER last_known_addr { host: "127.22.53.253" port: 45889 } attrs { promote: false } } peers { permanent_uuid: "07cfc39c8b0a4037bc133186d8f0c5aa" member_type: VOTER last_known_addr { host: "127.22.53.252" port: 39937 } attrs { promote: false } }
I20260926 07:30:22.638629 30186 leader_election.cc:288] T 00000000000000000000000000000000 P 5343811b11a34d0483a88e2adef7fc12 [CANDIDATE]: Term 3 election: Requested vote from peers b71a0534077f44f7b2cd8a7fa7688789 (127.22.53.254:44679), 07cfc39c8b0a4037bc133186d8f0c5aa (127.22.53.252:39937)
I20260926 07:30:22.638974 30187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:22.638727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:44679 (local address 127.0.0.1:45464)
0926 07:30:22.638863 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:22.638908 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:44679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260926 07:30:22.639146 29658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5343811b11a34d0483a88e2adef7fc12" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "07cfc39c8b0a4037bc133186d8f0c5aa"
I20260926 07:30:22.639295 29658 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 07cfc39c8b0a4037bc133186d8f0c5aa [term 2 FOLLOWER]: Advancing to term 3
I20260926 07:30:22.639756 30187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:22.639423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:44679 (local address 127.0.0.1:45468)
0926 07:30:22.639642 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:22.639697 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:44679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
W20260926 07:30:22.639933 29449 leader_election.cc:334] T 00000000000000000000000000000000 P 5343811b11a34d0483a88e2adef7fc12 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer b71a0534077f44f7b2cd8a7fa7688789 (127.22.53.254:44679): Network error: Client connection negotiation failed: client connection to 127.22.53.254:44679: connect: Connection refused (error 111)
I20260926 07:30:22.640674 29658 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 07cfc39c8b0a4037bc133186d8f0c5aa [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5343811b11a34d0483a88e2adef7fc12 in term 3.
I20260926 07:30:22.640956 29450 leader_election.cc:302] T 00000000000000000000000000000000 P 5343811b11a34d0483a88e2adef7fc12 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 07cfc39c8b0a4037bc133186d8f0c5aa, 5343811b11a34d0483a88e2adef7fc12; no voters: b71a0534077f44f7b2cd8a7fa7688789
I20260926 07:30:22.641196 30186 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5343811b11a34d0483a88e2adef7fc12 [term 3 FOLLOWER]: Leader election won for term 3
I20260926 07:30:22.641435 30186 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5343811b11a34d0483a88e2adef7fc12 [term 3 LEADER]: Becoming Leader. State: Replica: 5343811b11a34d0483a88e2adef7fc12, State: Running, Role: LEADER
I20260926 07:30:22.641728 30186 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5343811b11a34d0483a88e2adef7fc12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b71a0534077f44f7b2cd8a7fa7688789" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 44679 } } peers { permanent_uuid: "5343811b11a34d0483a88e2adef7fc12" member_type: VOTER last_known_addr { host: "127.22.53.253" port: 45889 } attrs { promote: false } } peers { permanent_uuid: "07cfc39c8b0a4037bc133186d8f0c5aa" member_type: VOTER last_known_addr { host: "127.22.53.252" port: 39937 } attrs { promote: false } }
I20260926 07:30:22.642940 30189 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5343811b11a34d0483a88e2adef7fc12 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5343811b11a34d0483a88e2adef7fc12. Latest consensus state: current_term: 3 leader_uuid: "5343811b11a34d0483a88e2adef7fc12" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b71a0534077f44f7b2cd8a7fa7688789" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 44679 } } peers { permanent_uuid: "5343811b11a34d0483a88e2adef7fc12" member_type: VOTER last_known_addr { host: "127.22.53.253" port: 45889 } attrs { promote: false } } peers { permanent_uuid: "07cfc39c8b0a4037bc133186d8f0c5aa" member_type: VOTER last_known_addr { host: "127.22.53.252" port: 39937 } attrs { promote: false } } }
I20260926 07:30:22.643090 30189 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5343811b11a34d0483a88e2adef7fc12 [sys.catalog]: This master's current role is: LEADER
I20260926 07:30:22.643371 30191 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:30:22.645824 30191 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:30:22.646245 30191 catalog_manager.cc:1259] Loaded cluster ID: 8a78e9ea9d25425092fd72f238f444f5
I20260926 07:30:22.646314 30191 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:30:22.646886 30191 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:30:22.647296 30191 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5343811b11a34d0483a88e2adef7fc12: Loaded TSK: 0
I20260926 07:30:22.648000 30191 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:30:22.696729 29464 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34610:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:30:22.699942 29464 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:30:22.703469 30187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:22.703104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:44679 (local address 127.0.0.1:45484)
0926 07:30:22.703297 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:22.703369 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:44679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260926 07:30:22.703493 29658 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 07cfc39c8b0a4037bc133186d8f0c5aa [term 3 FOLLOWER]: Refusing update from remote peer 5343811b11a34d0483a88e2adef7fc12: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260926 07:30:22.704102 30189 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5343811b11a34d0483a88e2adef7fc12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07cfc39c8b0a4037bc133186d8f0c5aa" member_type: VOTER last_known_addr { host: "127.22.53.252" port: 39937 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260926 07:30:22.704373 30187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:22.704103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:44679 (local address 127.0.0.1:45496)
0926 07:30:22.704240 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:22.704303 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:44679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260926 07:30:22.704596 29449 consensus_peers.cc:487] T 00000000000000000000000000000000 P 5343811b11a34d0483a88e2adef7fc12 -> Peer b71a0534077f44f7b2cd8a7fa7688789 (127.22.53.254:44679): Couldn't send request to peer b71a0534077f44f7b2cd8a7fa7688789. Status: Network error: Client connection negotiation failed: client connection to 127.22.53.254:44679: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 07:30:22.706097 30193 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07cfc39c8b0a4037bc133186d8f0c5aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5343811b11a34d0483a88e2adef7fc12. Latest consensus state: current_term: 3 leader_uuid: "5343811b11a34d0483a88e2adef7fc12" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b71a0534077f44f7b2cd8a7fa7688789" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 44679 } } peers { permanent_uuid: "5343811b11a34d0483a88e2adef7fc12" member_type: VOTER last_known_addr { host: "127.22.53.253" port: 45889 } attrs { promote: false } } peers { permanent_uuid: "07cfc39c8b0a4037bc133186d8f0c5aa" member_type: VOTER last_known_addr { host: "127.22.53.252" port: 39937 } attrs { promote: false } } }
I20260926 07:30:22.706301 30193 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07cfc39c8b0a4037bc133186d8f0c5aa [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:30:22.707414 30192 mvcc.cc:205] Tried to move back new op lower bound from 7333510441788002304 to 7333510441543548928. Current Snapshot: MvccSnapshot[applied={T|T < 7333510441788002304}]
I20260926 07:30:22.707688 30189 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5343811b11a34d0483a88e2adef7fc12 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "5343811b11a34d0483a88e2adef7fc12" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b71a0534077f44f7b2cd8a7fa7688789" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 44679 } } peers { permanent_uuid: "5343811b11a34d0483a88e2adef7fc12" member_type: VOTER last_known_addr { host: "127.22.53.253" port: 45889 } attrs { promote: false } } peers { permanent_uuid: "07cfc39c8b0a4037bc133186d8f0c5aa" member_type: VOTER last_known_addr { host: "127.22.53.252" port: 39937 } attrs { promote: false } } }
I20260926 07:30:22.707864 30189 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5343811b11a34d0483a88e2adef7fc12 [sys.catalog]: This master's current role is: LEADER
I20260926 07:30:22.708317 30193 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07cfc39c8b0a4037bc133186d8f0c5aa [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "5343811b11a34d0483a88e2adef7fc12" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b71a0534077f44f7b2cd8a7fa7688789" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 44679 } } peers { permanent_uuid: "5343811b11a34d0483a88e2adef7fc12" member_type: VOTER last_known_addr { host: "127.22.53.253" port: 45889 } attrs { promote: false } } peers { permanent_uuid: "07cfc39c8b0a4037bc133186d8f0c5aa" member_type: VOTER last_known_addr { host: "127.22.53.252" port: 39937 } attrs { promote: false } } }
I20260926 07:30:22.708565 30193 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07cfc39c8b0a4037bc133186d8f0c5aa [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:30:22.719192 30187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:22.714733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.194:38993 (local address 127.0.0.1:45604)
0926 07:30:22.714909 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:22.714925 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:22.715036 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:22.715503 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:22.715506 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:22.715526 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:30:22.715654 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:22.715663 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:22.716691 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:22.716694 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:22.717818 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:22.717829 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:22.718740 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:22.718744 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:22.718851 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:22.718878 (+ 27us) client_negotiation.cc:769] Sending connection context
0926 07:30:22.718952 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 07:30:22.719028 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87,"mutex_wait_us":52}
I20260926 07:30:22.719245 30199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:22.714898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45604 (local address 127.22.53.194:38993)
0926 07:30:22.715258 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:22.715262 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:22.715277 (+ 15us) server_negotiation.cc:407] Connection header received
0926 07:30:22.715330 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:22.715333 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:22.715387 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:22.715478 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:30:22.715757 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:22.716555 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:22.717959 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:22.718619 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:22.718733 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:22.718805 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:22.718992 (+ 187us) server_negotiation.cc:299] Negotiation successful
0926 07:30:22.719071 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":113,"threads_started":1}
I20260926 07:30:22.719941 30200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:22.715051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.195:42015 (local address 127.0.0.1:38746)
0926 07:30:22.715509 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:22.715523 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:22.715598 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:22.716159 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:22.716162 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:22.716179 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:30:22.716266 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:22.716271 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:22.717427 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:22.717430 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:22.718509 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:22.718516 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:22.719498 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:22.719501 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:22.719608 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:22.719643 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 07:30:22.719728 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 07:30:22.719804 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"mutex_wait_us":1,"thread_start_us":115,"threads_started":1}
I20260926 07:30:22.720008 30201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:22.715243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38746 (local address 127.22.53.195:42015)
0926 07:30:22.715848 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:22.715852 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:22.715866 (+ 14us) server_negotiation.cc:407] Connection header received
0926 07:30:22.715917 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:22.715919 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:22.715965 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:22.716051 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:30:22.716384 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:22.717308 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:22.718620 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:22.719275 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:22.719379 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:22.719440 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:22.719772 (+ 332us) server_negotiation.cc:299] Negotiation successful
0926 07:30:22.719861 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":76,"threads_started":1}
I20260926 07:30:22.720815 30035 tablet_service.cc:1467] Processing CreateTablet for tablet 90d04c07fc304c3c9024f0b4d292b264 (DEFAULT_TABLE table=test-table [id=f62708be9ee94c1fabda5375205bd9db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:30:22.721024 29899 tablet_service.cc:1467] Processing CreateTablet for tablet 90d04c07fc304c3c9024f0b4d292b264 (DEFAULT_TABLE table=test-table [id=f62708be9ee94c1fabda5375205bd9db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:30:22.721149 30035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90d04c07fc304c3c9024f0b4d292b264. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:30:22.721311 29899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90d04c07fc304c3c9024f0b4d292b264. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:30:22.721637 30203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:22.715419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.193:35569 (local address 127.0.0.1:53580)
0926 07:30:22.716150 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:22.716165 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:22.716246 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:22.718133 (+ 1887us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:22.718136 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:22.718158 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:30:22.718237 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:22.718242 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:22.719198 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:22.719201 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:22.720285 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:22.720298 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:22.721197 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:22.721201 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:22.721297 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:22.721323 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 07:30:22.721409 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 07:30:22.721484 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":654,"mutex_wait_us":324,"thread_start_us":107,"threads_started":1}
I20260926 07:30:22.721705 30202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:22.715550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53580 (local address 127.22.53.193:35569)
0926 07:30:22.717856 (+ 2306us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:22.717860 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:22.717871 (+ 11us) server_negotiation.cc:407] Connection header received
0926 07:30:22.717965 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:22.717967 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:22.718019 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:22.718113 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:30:22.718355 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:22.719061 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:22.720428 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:22.721062 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:22.721185 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:22.721249 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:22.721446 (+ 197us) server_negotiation.cc:299] Negotiation successful
0926 07:30:22.721556 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2230,"thread_start_us":104,"threads_started":1}
I20260926 07:30:22.723570 30205 tablet_bootstrap.cc:492] T 90d04c07fc304c3c9024f0b4d292b264 P ddbc2223fead4ddfa34d0a137efe01cb: Bootstrap starting.
I20260926 07:30:22.724253 29762 tablet_service.cc:1467] Processing CreateTablet for tablet 90d04c07fc304c3c9024f0b4d292b264 (DEFAULT_TABLE table=test-table [id=f62708be9ee94c1fabda5375205bd9db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:30:22.724503 29762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90d04c07fc304c3c9024f0b4d292b264. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:30:22.724522 30205 tablet_bootstrap.cc:654] T 90d04c07fc304c3c9024f0b4d292b264 P ddbc2223fead4ddfa34d0a137efe01cb: Neither blocks nor log segments found. Creating new log.
I20260926 07:30:22.724786 30204 tablet_bootstrap.cc:492] T 90d04c07fc304c3c9024f0b4d292b264 P ebec0858532b452191c80c0652eb9b8e: Bootstrap starting.
I20260926 07:30:22.725708 30204 tablet_bootstrap.cc:654] T 90d04c07fc304c3c9024f0b4d292b264 P ebec0858532b452191c80c0652eb9b8e: Neither blocks nor log segments found. Creating new log.
I20260926 07:30:22.726022 30205 tablet_bootstrap.cc:492] T 90d04c07fc304c3c9024f0b4d292b264 P ddbc2223fead4ddfa34d0a137efe01cb: No bootstrap required, opened a new log
I20260926 07:30:22.726126 30205 ts_tablet_manager.cc:1397] T 90d04c07fc304c3c9024f0b4d292b264 P ddbc2223fead4ddfa34d0a137efe01cb: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 07:30:22.726600 30205 raft_consensus.cc:348] T 90d04c07fc304c3c9024f0b4d292b264 P ddbc2223fead4ddfa34d0a137efe01cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddbc2223fead4ddfa34d0a137efe01cb" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 38993 } } peers { permanent_uuid: "ebec0858532b452191c80c0652eb9b8e" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 42015 } } peers { permanent_uuid: "96b38667cb7946819bcb4dfc480f93be" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 35569 } }
I20260926 07:30:22.726744 30205 raft_consensus.cc:374] T 90d04c07fc304c3c9024f0b4d292b264 P ddbc2223fead4ddfa34d0a137efe01cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:30:22.726778 30205 raft_consensus.cc:729] T 90d04c07fc304c3c9024f0b4d292b264 P ddbc2223fead4ddfa34d0a137efe01cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddbc2223fead4ddfa34d0a137efe01cb, State: Initialized, Role: FOLLOWER
I20260926 07:30:22.726915 30205 consensus_queue.cc:260] T 90d04c07fc304c3c9024f0b4d292b264 P ddbc2223fead4ddfa34d0a137efe01cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddbc2223fead4ddfa34d0a137efe01cb" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 38993 } } peers { permanent_uuid: "ebec0858532b452191c80c0652eb9b8e" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 42015 } } peers { permanent_uuid: "96b38667cb7946819bcb4dfc480f93be" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 35569 } }
I20260926 07:30:22.727301 30205 ts_tablet_manager.cc:1428] T 90d04c07fc304c3c9024f0b4d292b264 P ddbc2223fead4ddfa34d0a137efe01cb: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260926 07:30:22.727891 30204 tablet_bootstrap.cc:492] T 90d04c07fc304c3c9024f0b4d292b264 P ebec0858532b452191c80c0652eb9b8e: No bootstrap required, opened a new log
I20260926 07:30:22.728060 30204 ts_tablet_manager.cc:1397] T 90d04c07fc304c3c9024f0b4d292b264 P ebec0858532b452191c80c0652eb9b8e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 07:30:22.728103 30209 tablet_bootstrap.cc:492] T 90d04c07fc304c3c9024f0b4d292b264 P 96b38667cb7946819bcb4dfc480f93be: Bootstrap starting.
I20260926 07:30:22.728302 30185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:22.727844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:44679 (local address 127.22.53.194:59073)
0926 07:30:22.728159 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:22.728222 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:44679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228}
I20260926 07:30:22.728657 30204 raft_consensus.cc:348] T 90d04c07fc304c3c9024f0b4d292b264 P ebec0858532b452191c80c0652eb9b8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddbc2223fead4ddfa34d0a137efe01cb" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 38993 } } peers { permanent_uuid: "ebec0858532b452191c80c0652eb9b8e" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 42015 } } peers { permanent_uuid: "96b38667cb7946819bcb4dfc480f93be" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 35569 } }
I20260926 07:30:22.728842 30204 raft_consensus.cc:374] T 90d04c07fc304c3c9024f0b4d292b264 P ebec0858532b452191c80c0652eb9b8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:30:22.728926 30204 raft_consensus.cc:729] T 90d04c07fc304c3c9024f0b4d292b264 P ebec0858532b452191c80c0652eb9b8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebec0858532b452191c80c0652eb9b8e, State: Initialized, Role: FOLLOWER
I20260926 07:30:22.729208 30185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:22.728919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:44679 (local address 127.22.53.194:53871)
0926 07:30:22.729088 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:22.729138 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:44679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260926 07:30:22.729154 30204 consensus_queue.cc:260] T 90d04c07fc304c3c9024f0b4d292b264 P ebec0858532b452191c80c0652eb9b8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddbc2223fead4ddfa34d0a137efe01cb" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 38993 } } peers { permanent_uuid: "ebec0858532b452191c80c0652eb9b8e" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 42015 } } peers { permanent_uuid: "96b38667cb7946819bcb4dfc480f93be" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 35569 } }
I20260926 07:30:22.729516 30209 tablet_bootstrap.cc:654] T 90d04c07fc304c3c9024f0b4d292b264 P 96b38667cb7946819bcb4dfc480f93be: Neither blocks nor log segments found. Creating new log.
I20260926 07:30:22.730438 30183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:22.730179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:44679 (local address 127.22.53.195:33287)
0926 07:30:22.730306 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:22.730363 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:44679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260926 07:30:22.732365 30204 ts_tablet_manager.cc:1428] T 90d04c07fc304c3c9024f0b4d292b264 P ebec0858532b452191c80c0652eb9b8e: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20260926 07:30:22.732375 29967 heartbeater.cc:503] Master 127.22.53.253:45889 was elected leader, sending a full tablet report...
I20260926 07:30:22.732746 30103 heartbeater.cc:503] Master 127.22.53.253:45889 was elected leader, sending a full tablet report...
I20260926 07:30:22.733170 30183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:22.732849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:44679 (local address 127.22.53.195:59303)
0926 07:30:22.733034 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:22.733092 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:44679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260926 07:30:22.734846 30209 tablet_bootstrap.cc:492] T 90d04c07fc304c3c9024f0b4d292b264 P 96b38667cb7946819bcb4dfc480f93be: No bootstrap required, opened a new log
I20260926 07:30:22.735004 30209 ts_tablet_manager.cc:1397] T 90d04c07fc304c3c9024f0b4d292b264 P 96b38667cb7946819bcb4dfc480f93be: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260926 07:30:22.735457 30209 raft_consensus.cc:348] T 90d04c07fc304c3c9024f0b4d292b264 P 96b38667cb7946819bcb4dfc480f93be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddbc2223fead4ddfa34d0a137efe01cb" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 38993 } } peers { permanent_uuid: "ebec0858532b452191c80c0652eb9b8e" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 42015 } } peers { permanent_uuid: "96b38667cb7946819bcb4dfc480f93be" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 35569 } }
I20260926 07:30:22.735585 30209 raft_consensus.cc:374] T 90d04c07fc304c3c9024f0b4d292b264 P 96b38667cb7946819bcb4dfc480f93be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:30:22.735630 30209 raft_consensus.cc:729] T 90d04c07fc304c3c9024f0b4d292b264 P 96b38667cb7946819bcb4dfc480f93be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96b38667cb7946819bcb4dfc480f93be, State: Initialized, Role: FOLLOWER
I20260926 07:30:22.735771 30209 consensus_queue.cc:260] T 90d04c07fc304c3c9024f0b4d292b264 P 96b38667cb7946819bcb4dfc480f93be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddbc2223fead4ddfa34d0a137efe01cb" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 38993 } } peers { permanent_uuid: "ebec0858532b452191c80c0652eb9b8e" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 42015 } } peers { permanent_uuid: "96b38667cb7946819bcb4dfc480f93be" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 35569 } }
I20260926 07:30:22.736596 30209 ts_tablet_manager.cc:1428] T 90d04c07fc304c3c9024f0b4d292b264 P 96b38667cb7946819bcb4dfc480f93be: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 07:30:22.736945 30184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:22.736586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:44679 (local address 127.22.53.193:36265)
0926 07:30:22.736806 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:22.736867 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:44679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144}
I20260926 07:30:22.737672 30184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:22.737423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:44679 (local address 127.22.53.193:43239)
0926 07:30:22.737553 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:22.737610 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:44679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260926 07:30:22.737749 29830 heartbeater.cc:503] Master 127.22.53.253:45889 was elected leader, sending a full tablet report...
I20260926 07:30:22.886724 30212 raft_consensus.cc:482] T 90d04c07fc304c3c9024f0b4d292b264 P 96b38667cb7946819bcb4dfc480f93be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:30:22.886888 30212 raft_consensus.cc:504] T 90d04c07fc304c3c9024f0b4d292b264 P 96b38667cb7946819bcb4dfc480f93be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddbc2223fead4ddfa34d0a137efe01cb" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 38993 } } peers { permanent_uuid: "ebec0858532b452191c80c0652eb9b8e" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 42015 } } peers { permanent_uuid: "96b38667cb7946819bcb4dfc480f93be" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 35569 } }
I20260926 07:30:22.887956 30212 leader_election.cc:288] T 90d04c07fc304c3c9024f0b4d292b264 P 96b38667cb7946819bcb4dfc480f93be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ddbc2223fead4ddfa34d0a137efe01cb (127.22.53.194:38993), ebec0858532b452191c80c0652eb9b8e (127.22.53.195:42015)
W20260926 07:30:22.889974 30104 tablet.cc:2369] T 90d04c07fc304c3c9024f0b4d292b264 P ebec0858532b452191c80c0652eb9b8e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:30:22.892661 30184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:22.888278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.194:38993 (local address 127.22.53.193:51507)
0926 07:30:22.888423 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:22.888441 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:22.888594 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:22.889166 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:22.889169 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:22.889190 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:30:22.889331 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:22.889338 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:22.890350 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:22.890357 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:22.891325 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:22.891336 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:22.892083 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:22.892086 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:22.892243 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:22.892268 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 07:30:22.892363 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 07:30:22.892468 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":31}
I20260926 07:30:22.892645 30199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:22.888389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.193:51507 (local address 127.22.53.194:38993)
0926 07:30:22.888588 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:22.888593 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:22.888617 (+ 24us) server_negotiation.cc:407] Connection header received
0926 07:30:22.888738 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:22.888742 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:22.888793 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:22.888900 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:30:22.889458 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:22.890227 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:22.891473 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:22.891926 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:22.892078 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:22.892141 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:22.892380 (+ 239us) server_negotiation.cc:299] Negotiation successful
0926 07:30:22.892468 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 07:30:22.892645 30201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:22.888395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.193:44341 (local address 127.22.53.195:42015)
0926 07:30:22.888583 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:22.888588 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:22.889054 (+ 466us) server_negotiation.cc:407] Connection header received
0926 07:30:22.889203 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:22.889207 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:22.889251 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:22.889345 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:30:22.889672 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:22.890234 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:22.891504 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:22.891923 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:22.892077 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:22.892141 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:22.892394 (+ 253us) server_negotiation.cc:299] Negotiation successful
0926 07:30:22.892482 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 07:30:22.892684 30213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:22.888286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.195:42015 (local address 127.22.53.193:44341)
0926 07:30:22.888960 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:22.888976 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:22.889096 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:22.889353 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:22.889356 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:22.889371 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:30:22.889521 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:22.889528 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:22.890387 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:22.890390 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:22.891363 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:22.891372 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:22.892078 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:22.892082 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:22.892261 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:22.892279 (+ 18us) client_negotiation.cc:769] Sending connection context
0926 07:30:22.892374 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 07:30:22.892481 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"mutex_wait_us":474,"thread_start_us":139,"threads_started":1}
I20260926 07:30:22.893327 30055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90d04c07fc304c3c9024f0b4d292b264" candidate_uuid: "96b38667cb7946819bcb4dfc480f93be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebec0858532b452191c80c0652eb9b8e" is_pre_election: true
I20260926 07:30:22.893337 29919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90d04c07fc304c3c9024f0b4d292b264" candidate_uuid: "96b38667cb7946819bcb4dfc480f93be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddbc2223fead4ddfa34d0a137efe01cb" is_pre_election: true
I20260926 07:30:22.893508 30055 raft_consensus.cc:2454] T 90d04c07fc304c3c9024f0b4d292b264 P ebec0858532b452191c80c0652eb9b8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96b38667cb7946819bcb4dfc480f93be in term 0.
I20260926 07:30:22.893568 29919 raft_consensus.cc:2454] T 90d04c07fc304c3c9024f0b4d292b264 P ddbc2223fead4ddfa34d0a137efe01cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96b38667cb7946819bcb4dfc480f93be in term 0.
I20260926 07:30:22.893937 29715 leader_election.cc:302] T 90d04c07fc304c3c9024f0b4d292b264 P 96b38667cb7946819bcb4dfc480f93be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96b38667cb7946819bcb4dfc480f93be, ebec0858532b452191c80c0652eb9b8e; no voters:
I20260926 07:30:22.894266 30212 raft_consensus.cc:2781] T 90d04c07fc304c3c9024f0b4d292b264 P 96b38667cb7946819bcb4dfc480f93be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:30:22.894330 30212 raft_consensus.cc:482] T 90d04c07fc304c3c9024f0b4d292b264 P 96b38667cb7946819bcb4dfc480f93be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:30:22.894357 30212 raft_consensus.cc:3037] T 90d04c07fc304c3c9024f0b4d292b264 P 96b38667cb7946819bcb4dfc480f93be [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:30:22.895430 30212 raft_consensus.cc:504] T 90d04c07fc304c3c9024f0b4d292b264 P 96b38667cb7946819bcb4dfc480f93be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddbc2223fead4ddfa34d0a137efe01cb" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 38993 } } peers { permanent_uuid: "ebec0858532b452191c80c0652eb9b8e" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 42015 } } peers { permanent_uuid: "96b38667cb7946819bcb4dfc480f93be" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 35569 } }
I20260926 07:30:22.895860 30212 leader_election.cc:288] T 90d04c07fc304c3c9024f0b4d292b264 P 96b38667cb7946819bcb4dfc480f93be [CANDIDATE]: Term 1 election: Requested vote from peers ddbc2223fead4ddfa34d0a137efe01cb (127.22.53.194:38993), ebec0858532b452191c80c0652eb9b8e (127.22.53.195:42015)
I20260926 07:30:22.896330 29919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90d04c07fc304c3c9024f0b4d292b264" candidate_uuid: "96b38667cb7946819bcb4dfc480f93be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddbc2223fead4ddfa34d0a137efe01cb"
I20260926 07:30:22.896304 30055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90d04c07fc304c3c9024f0b4d292b264" candidate_uuid: "96b38667cb7946819bcb4dfc480f93be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebec0858532b452191c80c0652eb9b8e"
I20260926 07:30:22.896497 29919 raft_consensus.cc:3037] T 90d04c07fc304c3c9024f0b4d292b264 P ddbc2223fead4ddfa34d0a137efe01cb [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:30:22.896507 30055 raft_consensus.cc:3037] T 90d04c07fc304c3c9024f0b4d292b264 P ebec0858532b452191c80c0652eb9b8e [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:30:22.897539 30055 raft_consensus.cc:2454] T 90d04c07fc304c3c9024f0b4d292b264 P ebec0858532b452191c80c0652eb9b8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96b38667cb7946819bcb4dfc480f93be in term 1.
I20260926 07:30:22.897531 29919 raft_consensus.cc:2454] T 90d04c07fc304c3c9024f0b4d292b264 P ddbc2223fead4ddfa34d0a137efe01cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96b38667cb7946819bcb4dfc480f93be in term 1.
I20260926 07:30:22.897886 29715 leader_election.cc:302] T 90d04c07fc304c3c9024f0b4d292b264 P 96b38667cb7946819bcb4dfc480f93be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96b38667cb7946819bcb4dfc480f93be, ebec0858532b452191c80c0652eb9b8e; no voters:
I20260926 07:30:22.898092 30212 raft_consensus.cc:2781] T 90d04c07fc304c3c9024f0b4d292b264 P 96b38667cb7946819bcb4dfc480f93be [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:30:22.898303 30212 raft_consensus.cc:686] T 90d04c07fc304c3c9024f0b4d292b264 P 96b38667cb7946819bcb4dfc480f93be [term 1 LEADER]: Becoming Leader. State: Replica: 96b38667cb7946819bcb4dfc480f93be, State: Running, Role: LEADER
I20260926 07:30:22.898607 30212 consensus_queue.cc:237] T 90d04c07fc304c3c9024f0b4d292b264 P 96b38667cb7946819bcb4dfc480f93be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddbc2223fead4ddfa34d0a137efe01cb" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 38993 } } peers { permanent_uuid: "ebec0858532b452191c80c0652eb9b8e" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 42015 } } peers { permanent_uuid: "96b38667cb7946819bcb4dfc480f93be" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 35569 } }
I20260926 07:30:22.899199 30213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:22.898915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:44679 (local address 127.22.53.193:51253)
0926 07:30:22.899051 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:22.899112 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:44679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260926 07:30:22.901274 29465 catalog_manager.cc:5495] T 90d04c07fc304c3c9024f0b4d292b264 P 96b38667cb7946819bcb4dfc480f93be reported cstate change: term changed from 0 to 1, leader changed from <none> to 96b38667cb7946819bcb4dfc480f93be (127.22.53.193). New cstate: current_term: 1 leader_uuid: "96b38667cb7946819bcb4dfc480f93be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddbc2223fead4ddfa34d0a137efe01cb" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 38993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ebec0858532b452191c80c0652eb9b8e" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 42015 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96b38667cb7946819bcb4dfc480f93be" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 35569 } health_report { overall_health: HEALTHY } } }
W20260926 07:30:22.906942 29968 tablet.cc:2369] T 90d04c07fc304c3c9024f0b4d292b264 P ddbc2223fead4ddfa34d0a137efe01cb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:30:22.919188 30202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:22.915102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53588 (local address 127.22.53.193:35569)
0926 07:30:22.915232 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:22.915236 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:22.915249 (+ 13us) server_negotiation.cc:407] Connection header received
0926 07:30:22.915386 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:22.915389 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:22.915451 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:22.915551 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:30:22.915790 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:22.916307 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:22.917315 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:22.917560 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:22.917655 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:22.917928 (+ 273us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:30:22.918439 (+ 511us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:30:22.918551 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:22.918990 (+ 439us) server_negotiation.cc:299] Negotiation successful
0926 07:30:22.919060 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 07:30:22.922530 30055 raft_consensus.cc:1261] T 90d04c07fc304c3c9024f0b4d292b264 P ebec0858532b452191c80c0652eb9b8e [term 1 FOLLOWER]: Refusing update from remote peer 96b38667cb7946819bcb4dfc480f93be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:30:22.922792 29919 raft_consensus.cc:1261] T 90d04c07fc304c3c9024f0b4d292b264 P ddbc2223fead4ddfa34d0a137efe01cb [term 1 FOLLOWER]: Refusing update from remote peer 96b38667cb7946819bcb4dfc480f93be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:30:22.923195 30214 consensus_queue.cc:1035] T 90d04c07fc304c3c9024f0b4d292b264 P 96b38667cb7946819bcb4dfc480f93be [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebec0858532b452191c80c0652eb9b8e" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 42015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:30:22.923434 30212 consensus_queue.cc:1035] T 90d04c07fc304c3c9024f0b4d292b264 P 96b38667cb7946819bcb4dfc480f93be [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddbc2223fead4ddfa34d0a137efe01cb" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 38993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:30:22.925551 30185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:22.925278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:44679 (local address 127.22.53.194:32997)
0926 07:30:22.925430 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:22.925474 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:44679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260926 07:30:22.925551 30183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:22.925019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:44679 (local address 127.22.53.195:49551)
0926 07:30:22.925397 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:22.925465 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:44679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":300}
I20260926 07:30:22.928848 30218 mvcc.cc:205] Tried to move back new op lower bound from 7333510442684751872 to 7333510442595348480. Current Snapshot: MvccSnapshot[applied={T|T < 7333510442684751872}]
I20260926 07:30:22.929683 30213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:22.929393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:44679 (local address 127.22.53.193:35611)
0926 07:30:22.929522 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:22.929597 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:44679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260926 07:30:22.929720 30183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:22.929393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:44679 (local address 127.22.53.195:52891)
0926 07:30:22.929573 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:22.929636 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:44679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260926 07:30:22.931927 30185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:22.931635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:44679 (local address 127.22.53.194:55489)
0926 07:30:22.931750 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:22.931813 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:44679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260926 07:30:22.932348 30213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:22.932005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:44679 (local address 127.22.53.193:33147)
0926 07:30:22.932209 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:22.932272 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:44679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130}
I20260926 07:30:22.940910 29464 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:30:22.941148 29464 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:30:22.945622 29464 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5343811b11a34d0483a88e2adef7fc12: Sending DeleteTablet for 3 replicas of tablet 90d04c07fc304c3c9024f0b4d292b264
I20260926 07:30:22.946432 29899 tablet_service.cc:1514] Processing DeleteTablet for tablet 90d04c07fc304c3c9024f0b4d292b264 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:30:22 UTC) from {username='oryx', principal='oryx/127.22.53.253@KRBTEST.COM'} at 127.0.0.1:45604
I20260926 07:30:22.946760 30035 tablet_service.cc:1514] Processing DeleteTablet for tablet 90d04c07fc304c3c9024f0b4d292b264 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:30:22 UTC) from {username='oryx', principal='oryx/127.22.53.253@KRBTEST.COM'} at 127.0.0.1:38746
I20260926 07:30:22.946777 29762 tablet_service.cc:1514] Processing DeleteTablet for tablet 90d04c07fc304c3c9024f0b4d292b264 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:30:22 UTC) from {username='oryx', principal='oryx/127.22.53.253@KRBTEST.COM'} at 127.0.0.1:53580
I20260926 07:30:22.946897 30227 tablet_replica.cc:330] T 90d04c07fc304c3c9024f0b4d292b264 P ddbc2223fead4ddfa34d0a137efe01cb: stopping tablet replica
I20260926 07:30:22.947048 30227 raft_consensus.cc:2229] T 90d04c07fc304c3c9024f0b4d292b264 P ddbc2223fead4ddfa34d0a137efe01cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:30:22.947139 30228 tablet_replica.cc:330] T 90d04c07fc304c3c9024f0b4d292b264 P ebec0858532b452191c80c0652eb9b8e: stopping tablet replica
I20260926 07:30:22.947222 30227 raft_consensus.cc:2258] T 90d04c07fc304c3c9024f0b4d292b264 P ddbc2223fead4ddfa34d0a137efe01cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:30:22.947321 30228 raft_consensus.cc:2229] T 90d04c07fc304c3c9024f0b4d292b264 P ebec0858532b452191c80c0652eb9b8e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:30:22.947521 30228 raft_consensus.cc:2258] T 90d04c07fc304c3c9024f0b4d292b264 P ebec0858532b452191c80c0652eb9b8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:30:22.948065 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 29609
I20260926 07:30:22.948127 30227 ts_tablet_manager.cc:1905] T 90d04c07fc304c3c9024f0b4d292b264 P ddbc2223fead4ddfa34d0a137efe01cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:30:22.949534 30228 ts_tablet_manager.cc:1905] T 90d04c07fc304c3c9024f0b4d292b264 P ebec0858532b452191c80c0652eb9b8e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:30:22.951617 30227 ts_tablet_manager.cc:1918] T 90d04c07fc304c3c9024f0b4d292b264 P ddbc2223fead4ddfa34d0a137efe01cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:30:22.951713 30227 log.cc:1196] T 90d04c07fc304c3c9024f0b4d292b264 P ddbc2223fead4ddfa34d0a137efe01cb: Deleting WAL directory at /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/ts-1/wal/wals/90d04c07fc304c3c9024f0b4d292b264
I20260926 07:30:22.952000 30227 ts_tablet_manager.cc:1939] T 90d04c07fc304c3c9024f0b4d292b264 P ddbc2223fead4ddfa34d0a137efe01cb: Deleting consensus metadata
I20260926 07:30:22.952468 30228 ts_tablet_manager.cc:1918] T 90d04c07fc304c3c9024f0b4d292b264 P ebec0858532b452191c80c0652eb9b8e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:30:22.952590 30228 log.cc:1196] T 90d04c07fc304c3c9024f0b4d292b264 P ebec0858532b452191c80c0652eb9b8e: Deleting WAL directory at /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407783529717-22743-0/minicluster-data/ts-2/wal/wals/90d04c07fc304c3c9024f0b4d292b264
I20260926 07:30:22.952924 30228 ts_tablet_manager.cc:1939] T 90d04c07fc304c3c9024f0b4d292b264 P ebec0858532b452191c80c0652eb9b8e: Deleting consensus metadata
I20260926 07:30:22.953028 29452 catalog_manager.cc:4841] TS ddbc2223fead4ddfa34d0a137efe01cb (127.22.53.194:38993): tablet 90d04c07fc304c3c9024f0b4d292b264 (table test-table [id=f62708be9ee94c1fabda5375205bd9db]) successfully deleted
I20260926 07:30:22.953716 29449 catalog_manager.cc:4841] TS ebec0858532b452191c80c0652eb9b8e (127.22.53.195:42015): tablet 90d04c07fc304c3c9024f0b4d292b264 (table test-table [id=f62708be9ee94c1fabda5375205bd9db]) successfully deleted
I20260926 07:30:22.956266 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 29836
I20260926 07:30:22.957113 30203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:22.956446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.193:35569 (local address 127.0.0.1:53600)
0926 07:30:22.956931 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:22.957001 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.193:35569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375}
W20260926 07:30:22.957535 29449 catalog_manager.cc:4584] TS 96b38667cb7946819bcb4dfc480f93be (127.22.53.193:35569): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 90d04c07fc304c3c9024f0b4d292b264: Network error: Client connection negotiation failed: client connection to 127.22.53.193:35569: connect: Connection refused (error 111)
I20260926 07:30:22.964012 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 29972
I20260926 07:30:22.971271 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 29328
I20260926 07:30:22.978155 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 29431
2026-09-26T07:30:22Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7524 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407783529717-22743-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:30:23 dist-test-slave-wdjc krb5kdc[30232](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:30:23 dist-test-slave-wdjc krb5kdc[30232](info): set up 2 sockets
Sep 26 07:30:23 dist-test-slave-wdjc krb5kdc[30232](info): commencing operation
krb5kdc: starting...
W20260926 07:30:24.058007 22743 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407783529717-22743-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407783529717-22743-0/krb5kdc/test-user.keytab.
Sep 26 07:30:24 dist-test-slave-wdjc krb5kdc[30232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407824, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:30:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:30:24Z Disabled control of system clock
WARNING: no policy specified for oryx/127.22.53.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.53.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.53.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407783529717-22743-0/krb5kdc/oryx_127.22.53.254.keytab.
Entry for principal oryx/127.22.53.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407783529717-22743-0/krb5kdc/oryx_127.22.53.254.keytab.
I20260926 07:30:24.199494 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407783529717-22743-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407783529717-22743-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407783529717-22743-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.53.254:39267
--webserver_interface=127.22.53.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407783529717-22743-0/krb5kdc/oryx_127.22.53.254.keytab
--principal=oryx/127.22.53.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:41479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.53.254:39267
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:30:24.318485 30245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:30:24.318852 30245 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:30:24.318930 30245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:30:24.323230 30245 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:30:24.323351 30245 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 07:30:24.323381 30245 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:30:24.323426 30245 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:30:24.323493 30245 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:30:24.323529 30245 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:30:24.329499 30245 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:41479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407783529717-22743-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407783529717-22743-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.53.254:39267
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407783529717-22743-0/krb5kdc/oryx_127.22.53.254.keytab
--principal=oryx/127.22.53.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.53.254:39267
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.53.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30245
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407783529717-22743-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:30:24.330812 30245 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:30:24.331945 30245 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 07:30:24.341351 30245 server_base.cc:1034] running on GCE node
W20260926 07:30:24.341310 30251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:24.341321 30250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:24.341306 30253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:24.342343 30245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:30:24.343616 30245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:30:24.344797 30245 hybrid_clock.cc:648] HybridClock initialized: now 1790407824344784 us; error 36 us; skew 500 ppm
Sep 26 07:30:24 dist-test-slave-wdjc krb5kdc[30232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407824, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.53.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:30:24.347949 30245 init.cc:377] Logged in from keytab as oryx/127.22.53.254@KRBTEST.COM (short username oryx)
I20260926 07:30:24.349084 30245 webserver.cc:460] Webserver started at http://127.22.53.254:44289/ using document root <none> and password file <none>
I20260926 07:30:24.349649 30245 fs_manager.cc:359] Metadata directory not provided
I20260926 07:30:24.349735 30245 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:30:24.349936 30245 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:30:24.352200 30245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407783529717-22743-0/minicluster-data/master-0/data/instance:
uuid: "945929f517c847deaee9a27a48db5aa9"
format_stamp: "Formatted at 2026-09-26 07:30:24 on dist-test-slave-wdjc"
server_key: "6a4b88b72455b2019e1ea5ca15429b24"
server_key_iv: "820229f512fd68875a80b7e6cfef53b4"
server_key_version: "encryptionkey@0"
I20260926 07:30:24.352816 30245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407783529717-22743-0/minicluster-data/master-0/wal/instance:
uuid: "945929f517c847deaee9a27a48db5aa9"
format_stamp: "Formatted at 2026-09-26 07:30:24 on dist-test-slave-wdjc"
server_key: "6a4b88b72455b2019e1ea5ca15429b24"
server_key_iv: "820229f512fd68875a80b7e6cfef53b4"
server_key_version: "encryptionkey@0"
I20260926 07:30:24.356962 30245 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 07:30:24.359684 30260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:24.361186 30245 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:30:24.361320 30245 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407783529717-22743-0/minicluster-data/master-0/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407783529717-22743-0/minicluster-data/master-0/wal
uuid: "945929f517c847deaee9a27a48db5aa9"
format_stamp: "Formatted at 2026-09-26 07:30:24 on dist-test-slave-wdjc"
server_key: "6a4b88b72455b2019e1ea5ca15429b24"
server_key_iv: "820229f512fd68875a80b7e6cfef53b4"
server_key_version: "encryptionkey@0"
I20260926 07:30:24.361435 30245 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407783529717-22743-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407783529717-22743-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407783529717-22743-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:30:24.373904 30245 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:30:24.377342 30245 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:30:24.377513 30245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:30:24.386117 30245 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.254:39267
I20260926 07:30:24.386142 30322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.254:39267 every 8 connection(s)
I20260926 07:30:24.387178 30245 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
I20260926 07:30:24.390236 30323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:30:24.396216 30323 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 945929f517c847deaee9a27a48db5aa9: Bootstrap starting.
I20260926 07:30:24.396893 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 30245
I20260926 07:30:24.397033 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407783529717-22743-0/minicluster-data/master-0/wal/instance
I20260926 07:30:24.397312 22743 external_mini_cluster.cc:1428] Setting key 4061a29d0e7f982bb4348fe03f68b10e
I20260926 07:30:24.399248 30323 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 945929f517c847deaee9a27a48db5aa9: Neither blocks nor log segments found. Creating new log.
I20260926 07:30:24.400053 30323 log.cc:824] T 00000000000000000000000000000000 P 945929f517c847deaee9a27a48db5aa9: Log is configured to *not* fsync() on all Append() calls
I20260926 07:30:24.402633 30323 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 945929f517c847deaee9a27a48db5aa9: No bootstrap required, opened a new log
Sep 26 07:30:24 dist-test-slave-wdjc krb5kdc[30232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407824, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM
I20260926 07:30:24.407869 30323 raft_consensus.cc:348] T 00000000000000000000000000000000 P 945929f517c847deaee9a27a48db5aa9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "945929f517c847deaee9a27a48db5aa9" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 39267 } }
I20260926 07:30:24.408213 30323 raft_consensus.cc:374] T 00000000000000000000000000000000 P 945929f517c847deaee9a27a48db5aa9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:30:24.408306 30323 raft_consensus.cc:729] T 00000000000000000000000000000000 P 945929f517c847deaee9a27a48db5aa9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 945929f517c847deaee9a27a48db5aa9, State: Initialized, Role: FOLLOWER
I20260926 07:30:24.408861 30323 consensus_queue.cc:260] T 00000000000000000000000000000000 P 945929f517c847deaee9a27a48db5aa9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "945929f517c847deaee9a27a48db5aa9" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 39267 } }
I20260926 07:30:24.409027 30323 raft_consensus.cc:388] T 00000000000000000000000000000000 P 945929f517c847deaee9a27a48db5aa9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:30:24.409133 30323 raft_consensus.cc:482] T 00000000000000000000000000000000 P 945929f517c847deaee9a27a48db5aa9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:30:24.409240 30323 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 945929f517c847deaee9a27a48db5aa9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:30:24.410287 30323 raft_consensus.cc:504] T 00000000000000000000000000000000 P 945929f517c847deaee9a27a48db5aa9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "945929f517c847deaee9a27a48db5aa9" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 39267 } }
I20260926 07:30:24.410825 30323 leader_election.cc:302] T 00000000000000000000000000000000 P 945929f517c847deaee9a27a48db5aa9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 945929f517c847deaee9a27a48db5aa9; no voters:
I20260926 07:30:24.411279 30323 leader_election.cc:288] T 00000000000000000000000000000000 P 945929f517c847deaee9a27a48db5aa9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:30:24.411376 30328 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 945929f517c847deaee9a27a48db5aa9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:30:24.411626 30328 raft_consensus.cc:686] T 00000000000000000000000000000000 P 945929f517c847deaee9a27a48db5aa9 [term 1 LEADER]: Becoming Leader. State: Replica: 945929f517c847deaee9a27a48db5aa9, State: Running, Role: LEADER
I20260926 07:30:24.412006 30328 consensus_queue.cc:237] T 00000000000000000000000000000000 P 945929f517c847deaee9a27a48db5aa9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "945929f517c847deaee9a27a48db5aa9" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 39267 } }
I20260926 07:30:24.412828 30323 sys_catalog.cc:564] T 00000000000000000000000000000000 P 945929f517c847deaee9a27a48db5aa9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:30:24.413966 30330 sys_catalog.cc:455] T 00000000000000000000000000000000 P 945929f517c847deaee9a27a48db5aa9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 945929f517c847deaee9a27a48db5aa9. Latest consensus state: current_term: 1 leader_uuid: "945929f517c847deaee9a27a48db5aa9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "945929f517c847deaee9a27a48db5aa9" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 39267 } } }
I20260926 07:30:24.414100 30330 sys_catalog.cc:458] T 00000000000000000000000000000000 P 945929f517c847deaee9a27a48db5aa9 [sys.catalog]: This master's current role is: LEADER
I20260926 07:30:24.414237 30329 sys_catalog.cc:455] T 00000000000000000000000000000000 P 945929f517c847deaee9a27a48db5aa9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "945929f517c847deaee9a27a48db5aa9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "945929f517c847deaee9a27a48db5aa9" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 39267 } } }
I20260926 07:30:24.414419 30329 sys_catalog.cc:458] T 00000000000000000000000000000000 P 945929f517c847deaee9a27a48db5aa9 [sys.catalog]: This master's current role is: LEADER
I20260926 07:30:24.414628 30326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:24.398814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40500 (local address 127.22.53.254:39267)
0926 07:30:24.399273 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:24.399290 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:24.399327 (+ 37us) server_negotiation.cc:407] Connection header received
0926 07:30:24.400096 (+ 769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:24.400115 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:24.400618 (+ 503us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:24.401044 (+ 426us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:24.402062 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:24.403174 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:24.404437 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:24.405071 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:24.405222 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:24.408586 (+ 3364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:24.408596 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:24.408635 (+ 39us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:24.408682 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:24.410740 (+ 2058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:24.411357 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:24.411386 (+ 29us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:24.411393 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:24.411559 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:24.411936 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:24.411940 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:24.411943 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:24.412469 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:24.412697 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:24.413095 (+ 398us) server_negotiation.cc:299] Negotiation successful
0926 07:30:24.413446 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":160,"threads_started":1}
I20260926 07:30:24.414691 30337 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:30:24.418108 30337 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:30:24.424710 30337 catalog_manager.cc:1347] Generated new cluster ID: b4bca35f9e04469e8652990181190f3e
I20260926 07:30:24.424806 30337 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:30:24.431962 30337 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:30:24.433436 30337 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:30:24.439874 30337 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 945929f517c847deaee9a27a48db5aa9: Generated new TSK 0
I20260926 07:30:24.440840 30337 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.22.53.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.53.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.53.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407783529717-22743-0/krb5kdc/oryx_127.22.53.193.keytab.
Entry for principal oryx/127.22.53.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407783529717-22743-0/krb5kdc/oryx_127.22.53.193.keytab.
I20260926 07:30:24.472213 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407783529717-22743-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407783529717-22743-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407783529717-22743-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407783529717-22743-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.53.193:0
--local_ip_for_outbound_sockets=127.22.53.193
--webserver_interface=127.22.53.193
--webserver_port=0
--tserver_master_addrs=127.22.53.254:39267
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407783529717-22743-0/krb5kdc/oryx_127.22.53.193.keytab
--principal=oryx/127.22.53.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:41479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:30:24.594180 30349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:30:24.594619 30349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:30:24.594703 30349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:30:24.599594 30349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:30:24.599714 30349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:30:24.599797 30349 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 07:30:24.599887 30349 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.53.193
I20260926 07:30:24.606009 30349 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:41479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407783529717-22743-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407783529717-22743-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407783529717-22743-0/krb5kdc/oryx_127.22.53.193.keytab
--principal=oryx/127.22.53.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.53.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407783529717-22743-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.53.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.53.254:39267
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30349
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.53.193
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407783529717-22743-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:30:24.607223 30349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:30:24.608304 30349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:30:24.619414 30357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:24.619424 30354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:24.619424 30355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:24.619493 30349 server_base.cc:1034] running on GCE node
I20260926 07:30:24.620250 30349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:30:24.621063 30349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:30:24.622241 30349 hybrid_clock.cc:648] HybridClock initialized: now 1790407824622216 us; error 40 us; skew 500 ppm
Sep 26 07:30:24 dist-test-slave-wdjc krb5kdc[30232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407824, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.53.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:30:24.625412 30349 init.cc:377] Logged in from keytab as oryx/127.22.53.193@KRBTEST.COM (short username oryx)
I20260926 07:30:24.626506 30349 webserver.cc:460] Webserver started at http://127.22.53.193:43961/ using document root <none> and password file <none>
I20260926 07:30:24.627134 30349 fs_manager.cc:359] Metadata directory not provided
I20260926 07:30:24.627231 30349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:30:24.627481 30349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:30:24.629911 30349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407783529717-22743-0/minicluster-data/ts-0/data/instance:
uuid: "39b2209b30574a95bf503e074a4f7d29"
format_stamp: "Formatted at 2026-09-26 07:30:24 on dist-test-slave-wdjc"
server_key: "e4bfca90f42cf8ccf353f2c5dc991a58"
server_key_iv: "ee3ec14ab0351f7e0f243ebeb91f2091"
server_key_version: "encryptionkey@0"
I20260926 07:30:24.630545 30349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407783529717-22743-0/minicluster-data/ts-0/wal/instance:
uuid: "39b2209b30574a95bf503e074a4f7d29"
format_stamp: "Formatted at 2026-09-26 07:30:24 on dist-test-slave-wdjc"
server_key: "e4bfca90f42cf8ccf353f2c5dc991a58"
server_key_iv: "ee3ec14ab0351f7e0f243ebeb91f2091"
server_key_version: "encryptionkey@0"
I20260926 07:30:24.634773 30349 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 07:30:24.637789 30364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:24.639088 30349 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:30:24.639266 30349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407783529717-22743-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407783529717-22743-0/minicluster-data/ts-0/wal
uuid: "39b2209b30574a95bf503e074a4f7d29"
format_stamp: "Formatted at 2026-09-26 07:30:24 on dist-test-slave-wdjc"
server_key: "e4bfca90f42cf8ccf353f2c5dc991a58"
server_key_iv: "ee3ec14ab0351f7e0f243ebeb91f2091"
server_key_version: "encryptionkey@0"
I20260926 07:30:24.639410 30349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407783529717-22743-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407783529717-22743-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407783529717-22743-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:30:24.669502 30349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:30:24.673331 30349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:30:24.673570 30349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:30:24.675199 30349 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:30:24.675279 30349 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:24.675346 30349 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:30:24.675390 30349 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 07:30:24 dist-test-slave-wdjc krb5kdc[30232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407824, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.53.193@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM
I20260926 07:30:24.687418 30349 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.193:34137
I20260926 07:30:24.687587 30479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.193:34137 every 8 connection(s)
I20260926 07:30:24.688690 30349 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407783529717-22743-0/minicluster-data/ts-0/data/info.pb
I20260926 07:30:24.690510 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 30349
I20260926 07:30:24.690668 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407783529717-22743-0/minicluster-data/ts-0/wal/instance
I20260926 07:30:24.690977 22743 external_mini_cluster.cc:1428] Setting key ce95e0bade06d2e6d979d8eff6b33072
I20260926 07:30:24.691738 30326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:24.675985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.193:59613 (local address 127.22.53.254:39267)
0926 07:30:24.676145 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:24.676150 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:24.677271 (+ 1121us) server_negotiation.cc:407] Connection header received
0926 07:30:24.678653 (+ 1382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:24.678658 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:24.678736 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:24.678802 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:24.681074 (+ 2272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:24.681667 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:24.683101 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:24.683355 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:24.683492 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:24.686698 (+ 3206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:24.686705 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:24.686711 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:24.686750 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:24.688774 (+ 2024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:24.689613 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:24.689620 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:24.689622 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:24.689743 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:24.690228 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:24.690234 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:24.690235 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:24.690533 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:24.690683 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:24.691395 (+ 712us) server_negotiation.cc:299] Negotiation successful
0926 07:30:24.691561 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 07:30:24.692855 30373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:24.676452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:39267 (local address 127.22.53.193:59613)
0926 07:30:24.677056 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:24.677124 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:24.678326 (+ 1202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:24.679314 (+ 988us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:24.679328 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:24.679901 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:24.680825 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:24.680855 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:24.681847 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:24.681851 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:24.682932 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:24.682945 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:24.683564 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:24.683568 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:24.683754 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:24.684495 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:24.684515 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:24.686446 (+ 1931us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:30:24.688977 (+ 2531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:24.688984 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:24.689001 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:24.689445 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:24.689873 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:24.689876 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:24.689878 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:24.690037 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:24.690699 (+ 662us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:30:24.690708 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:30:24.691039 (+ 331us) client_negotiation.cc:769] Sending connection context
0926 07:30:24.691278 (+ 239us) client_negotiation.cc:240] Negotiation successful
0926 07:30:24.691698 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":118,"threads_started":1}
I20260926 07:30:24.694394 30480 heartbeater.cc:347] Connected to a master server at 127.22.53.254:39267
I20260926 07:30:24.694602 30480 heartbeater.cc:464] Registering TS with master...
I20260926 07:30:24.695348 30480 heartbeater.cc:511] Master 127.22.53.254:39267 requested a full tablet report, sending...
I20260926 07:30:24.697526 30276 ts_manager.cc:194] Registered new tserver with Master: 39b2209b30574a95bf503e074a4f7d29 (127.22.53.193:34137)
I20260926 07:30:24.699355 30276 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.53.193@KRBTEST.COM'} at 127.22.53.193:59613
WARNING: no policy specified for oryx/127.22.53.194@KRBTEST.COM; defaulting to no policy
I20260926 07:30:24.706138 30487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:24.697167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:39267 (local address 127.22.53.193:47589)
0926 07:30:24.697536 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:24.697550 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:24.697688 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:24.697969 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:24.697972 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:24.698209 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:24.698326 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:24.698332 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:24.699291 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:24.699295 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:24.699929 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:24.699939 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:24.700579 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:24.700587 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:24.700729 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:24.701434 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:24.701455 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:24.701969 (+ 514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:30:24.704079 (+ 2110us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:24.704083 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:24.704085 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:24.704424 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:24.704968 (+ 544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:24.704971 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:24.704973 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:24.705069 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:24.705569 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:30:24.705575 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:30:24.705690 (+ 115us) client_negotiation.cc:769] Sending connection context
0926 07:30:24.705797 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 07:30:24.705957 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"mutex_wait_us":1,"thread_start_us":134,"threads_started":1}
I20260926 07:30:24.706241 30326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:24.697213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.193:47589 (local address 127.22.53.254:39267)
0926 07:30:24.697360 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:24.697365 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:24.697634 (+ 269us) server_negotiation.cc:407] Connection header received
0926 07:30:24.697797 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:24.697800 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:24.697856 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:24.697953 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:24.698452 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:24.699148 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:24.700112 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:24.700372 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:24.700566 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:24.702147 (+ 1581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:24.702151 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:24.702154 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:24.702181 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:24.703923 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:24.704718 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:24.704721 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:24.704722 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:24.704831 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:24.705184 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:24.705186 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:24.705188 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:24.705421 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:24.705558 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:24.705907 (+ 349us) server_negotiation.cc:299] Negotiation successful
0926 07:30:24.706067 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":125,"server-negotiator.queue_time_us":52}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.53.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.53.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407783529717-22743-0/krb5kdc/oryx_127.22.53.194.keytab.
Entry for principal oryx/127.22.53.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407783529717-22743-0/krb5kdc/oryx_127.22.53.194.keytab.
I20260926 07:30:24.727871 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407783529717-22743-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407783529717-22743-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407783529717-22743-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407783529717-22743-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.53.194:0
--local_ip_for_outbound_sockets=127.22.53.194
--webserver_interface=127.22.53.194
--webserver_port=0
--tserver_master_addrs=127.22.53.254:39267
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407783529717-22743-0/krb5kdc/oryx_127.22.53.194.keytab
--principal=oryx/127.22.53.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:41479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:30:24.848960 30489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:30:24.849342 30489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:30:24.849428 30489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:30:24.853761 30489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:30:24.853890 30489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:30:24.853972 30489 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 07:30:24.854058 30489 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.53.194
I20260926 07:30:24.859941 30489 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:41479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407783529717-22743-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407783529717-22743-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407783529717-22743-0/krb5kdc/oryx_127.22.53.194.keytab
--principal=oryx/127.22.53.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.53.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407783529717-22743-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.53.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.53.254:39267
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30489
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.53.194
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407783529717-22743-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:30:24.861285 30489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:30:24.862407 30489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:30:24.873412 30494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:24.873422 30497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:24.873412 30495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:24.873682 30489 server_base.cc:1034] running on GCE node
I20260926 07:30:24.874363 30489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:30:24.875095 30489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:30:24.876264 30489 hybrid_clock.cc:648] HybridClock initialized: now 1790407824876250 us; error 35 us; skew 500 ppm
Sep 26 07:30:24 dist-test-slave-wdjc krb5kdc[30232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407824, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.53.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:30:24.879447 30489 init.cc:377] Logged in from keytab as oryx/127.22.53.194@KRBTEST.COM (short username oryx)
I20260926 07:30:24.880650 30489 webserver.cc:460] Webserver started at http://127.22.53.194:44397/ using document root <none> and password file <none>
I20260926 07:30:24.881302 30489 fs_manager.cc:359] Metadata directory not provided
I20260926 07:30:24.881409 30489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:30:24.881671 30489 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:30:24.884142 30489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407783529717-22743-0/minicluster-data/ts-1/data/instance:
uuid: "d3cd339e18e549a9b1b07935fe4dd0fc"
format_stamp: "Formatted at 2026-09-26 07:30:24 on dist-test-slave-wdjc"
server_key: "d7881434c29be9dea4c18a07ea2ed9c6"
server_key_iv: "cf1cb0cba9655dad35389fb29c80f41e"
server_key_version: "encryptionkey@0"
I20260926 07:30:24.884914 30489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407783529717-22743-0/minicluster-data/ts-1/wal/instance:
uuid: "d3cd339e18e549a9b1b07935fe4dd0fc"
format_stamp: "Formatted at 2026-09-26 07:30:24 on dist-test-slave-wdjc"
server_key: "d7881434c29be9dea4c18a07ea2ed9c6"
server_key_iv: "cf1cb0cba9655dad35389fb29c80f41e"
server_key_version: "encryptionkey@0"
I20260926 07:30:24.889678 30489 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 07:30:24.893087 30504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:24.894933 30489 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 07:30:24.895153 30489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407783529717-22743-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407783529717-22743-0/minicluster-data/ts-1/wal
uuid: "d3cd339e18e549a9b1b07935fe4dd0fc"
format_stamp: "Formatted at 2026-09-26 07:30:24 on dist-test-slave-wdjc"
server_key: "d7881434c29be9dea4c18a07ea2ed9c6"
server_key_iv: "cf1cb0cba9655dad35389fb29c80f41e"
server_key_version: "encryptionkey@0"
I20260926 07:30:24.895352 30489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407783529717-22743-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407783529717-22743-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407783529717-22743-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:30:24.909719 30489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:30:24.913542 30489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:30:24.913761 30489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:30:24.915575 30489 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:30:24.915655 30489 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:24.915730 30489 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:30:24.915783 30489 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 07:30:24 dist-test-slave-wdjc krb5kdc[30232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407824, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.53.194@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM
I20260926 07:30:24.927708 30489 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.194:38789
I20260926 07:30:24.927802 30619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.194:38789 every 8 connection(s)
I20260926 07:30:24.929061 30489 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407783529717-22743-0/minicluster-data/ts-1/data/info.pb
I20260926 07:30:24.931941 30326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:24.916486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.194:35163 (local address 127.22.53.254:39267)
0926 07:30:24.916660 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:24.916665 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:24.917676 (+ 1011us) server_negotiation.cc:407] Connection header received
0926 07:30:24.919196 (+ 1520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:24.919200 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:24.919266 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:24.919353 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:24.921118 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:24.921671 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:24.923002 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:24.923439 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:24.923569 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:24.927437 (+ 3868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:24.927443 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:24.927448 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:24.927491 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:24.929356 (+ 1865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:24.930150 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:24.930154 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:24.930156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:24.930263 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:24.930723 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:24.930726 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:24.930728 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:24.930978 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:24.931077 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:24.931678 (+ 601us) server_negotiation.cc:299] Negotiation successful
0926 07:30:24.931808 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 07:30:24.932960 30513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:24.917002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:39267 (local address 127.22.53.194:35163)
0926 07:30:24.917483 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:24.917536 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:24.918952 (+ 1416us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:24.919633 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:24.919641 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:24.920079 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:24.920945 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:24.920963 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:24.921831 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:24.921834 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:24.922816 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:24.922827 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:24.923653 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:24.923657 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:24.923898 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:24.925231 (+ 1333us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:24.925250 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:24.927119 (+ 1869us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:30:24.929559 (+ 2440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:24.929569 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:24.929586 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:24.929991 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:24.930381 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:24.930384 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:24.930385 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:24.930578 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:24.931125 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:30:24.931131 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:30:24.931410 (+ 279us) client_negotiation.cc:769] Sending connection context
0926 07:30:24.931576 (+ 166us) client_negotiation.cc:240] Negotiation successful
0926 07:30:24.931889 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":139,"threads_started":1}
I20260926 07:30:24.934562 30620 heartbeater.cc:347] Connected to a master server at 127.22.53.254:39267
I20260926 07:30:24.934737 30620 heartbeater.cc:464] Registering TS with master...
I20260926 07:30:24.935386 30620 heartbeater.cc:511] Master 127.22.53.254:39267 requested a full tablet report, sending...
I20260926 07:30:24.935912 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 30489
I20260926 07:30:24.936066 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407783529717-22743-0/minicluster-data/ts-1/wal/instance
I20260926 07:30:24.936396 22743 external_mini_cluster.cc:1428] Setting key fda23e1ee8b1c3f48eeba02dc004f3ec
I20260926 07:30:24.937320 30276 ts_manager.cc:194] Registered new tserver with Master: d3cd339e18e549a9b1b07935fe4dd0fc (127.22.53.194:38789)
I20260926 07:30:24.938076 30276 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.53.194@KRBTEST.COM'} at 127.22.53.194:35163
I20260926 07:30:24.947788 30626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:24.937327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:39267 (local address 127.22.53.194:53317)
0926 07:30:24.937996 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:24.938011 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:24.938134 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:24.938602 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:24.938608 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:24.938918 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:24.939106 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:24.939120 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:24.940643 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:24.940646 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:24.941266 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:24.941277 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:24.941983 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:24.941987 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:24.942105 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:24.942728 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:24.942741 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:24.943232 (+ 491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:30:24.945647 (+ 2415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:24.945650 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:24.945651 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:24.945893 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:24.946257 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:24.946259 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:24.946260 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:24.946341 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:24.947236 (+ 895us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:30:24.947239 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:30:24.947344 (+ 105us) client_negotiation.cc:769] Sending connection context
0926 07:30:24.947468 (+ 124us) client_negotiation.cc:240] Negotiation successful
0926 07:30:24.947625 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":144,"threads_started":1}
I20260926 07:30:24.947922 30326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:24.937416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.194:53317 (local address 127.22.53.254:39267)
0926 07:30:24.937565 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:24.937570 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:24.938085 (+ 515us) server_negotiation.cc:407] Connection header received
0926 07:30:24.938293 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:24.938296 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:24.938358 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:24.938457 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:24.939292 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:24.940217 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:24.941413 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:24.941823 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:24.941971 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:24.943413 (+ 1442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:24.943419 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:24.943424 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:24.943460 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:24.945493 (+ 2033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:24.946016 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:24.946021 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:24.946023 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:24.946145 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:24.946446 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:24.946451 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:24.946454 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:24.946820 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:24.947091 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:24.947584 (+ 493us) server_negotiation.cc:299] Negotiation successful
0926 07:30:24.947748 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
WARNING: no policy specified for oryx/127.22.53.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.53.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.53.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407783529717-22743-0/krb5kdc/oryx_127.22.53.195.keytab.
Entry for principal oryx/127.22.53.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407783529717-22743-0/krb5kdc/oryx_127.22.53.195.keytab.
I20260926 07:30:24.968317 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407783529717-22743-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407783529717-22743-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407783529717-22743-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407783529717-22743-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.53.195:0
--local_ip_for_outbound_sockets=127.22.53.195
--webserver_interface=127.22.53.195
--webserver_port=0
--tserver_master_addrs=127.22.53.254:39267
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407783529717-22743-0/krb5kdc/oryx_127.22.53.195.keytab
--principal=oryx/127.22.53.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:41479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:30:25.088492 30629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:30:25.088894 30629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:30:25.088944 30629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:30:25.093444 30629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:30:25.093568 30629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:30:25.093613 30629 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 07:30:25.093685 30629 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.53.195
I20260926 07:30:25.099740 30629 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:41479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407783529717-22743-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407783529717-22743-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407783529717-22743-0/krb5kdc/oryx_127.22.53.195.keytab
--principal=oryx/127.22.53.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.53.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407783529717-22743-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.53.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.53.254:39267
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30629
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.53.195
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407783529717-22743-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:30:25.101073 30629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:30:25.102161 30629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:30:25.112560 30634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:25.112609 30635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:25.112758 30629 server_base.cc:1034] running on GCE node
W20260926 07:30:25.112521 30637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:25.113426 30629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:30:25.114220 30629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:30:25.115453 30629 hybrid_clock.cc:648] HybridClock initialized: now 1790407825115412 us; error 65 us; skew 500 ppm
Sep 26 07:30:25 dist-test-slave-wdjc krb5kdc[30232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407825, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.53.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:30:25.118734 30629 init.cc:377] Logged in from keytab as oryx/127.22.53.195@KRBTEST.COM (short username oryx)
I20260926 07:30:25.119880 30629 webserver.cc:460] Webserver started at http://127.22.53.195:40267/ using document root <none> and password file <none>
I20260926 07:30:25.120603 30629 fs_manager.cc:359] Metadata directory not provided
I20260926 07:30:25.120704 30629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:30:25.120963 30629 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:30:25.123211 30629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407783529717-22743-0/minicluster-data/ts-2/data/instance:
uuid: "6d6bb899bd634ecd9fb3952a365d22ad"
format_stamp: "Formatted at 2026-09-26 07:30:25 on dist-test-slave-wdjc"
server_key: "c6563360d1be0b9223465a43f27af876"
server_key_iv: "26eb684878573da7daed314cb716db8e"
server_key_version: "encryptionkey@0"
I20260926 07:30:25.123831 30629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407783529717-22743-0/minicluster-data/ts-2/wal/instance:
uuid: "6d6bb899bd634ecd9fb3952a365d22ad"
format_stamp: "Formatted at 2026-09-26 07:30:25 on dist-test-slave-wdjc"
server_key: "c6563360d1be0b9223465a43f27af876"
server_key_iv: "26eb684878573da7daed314cb716db8e"
server_key_version: "encryptionkey@0"
I20260926 07:30:25.128091 30629 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 07:30:25.130982 30644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:25.132445 30629 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 07:30:25.132648 30629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407783529717-22743-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407783529717-22743-0/minicluster-data/ts-2/wal
uuid: "6d6bb899bd634ecd9fb3952a365d22ad"
format_stamp: "Formatted at 2026-09-26 07:30:25 on dist-test-slave-wdjc"
server_key: "c6563360d1be0b9223465a43f27af876"
server_key_iv: "26eb684878573da7daed314cb716db8e"
server_key_version: "encryptionkey@0"
I20260926 07:30:25.132789 30629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407783529717-22743-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407783529717-22743-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407783529717-22743-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:30:25.152395 30629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:30:25.156020 30629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:30:25.156265 30629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:30:25.157969 30629 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:30:25.158053 30629 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:25.158123 30629 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:30:25.158169 30629 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 07:30:25 dist-test-slave-wdjc krb5kdc[30232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407825, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.53.195@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM
I20260926 07:30:25.170205 30629 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.195:34909
I20260926 07:30:25.170277 30759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.195:34909 every 8 connection(s)
I20260926 07:30:25.171270 30629 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407783529717-22743-0/minicluster-data/ts-2/data/info.pb
I20260926 07:30:25.173769 30326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:25.158869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.195:49577 (local address 127.22.53.254:39267)
0926 07:30:25.159005 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:25.159010 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:25.160238 (+ 1228us) server_negotiation.cc:407] Connection header received
0926 07:30:25.161486 (+ 1248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:25.161489 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:25.161545 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:25.161622 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:25.163459 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:25.164422 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:25.165995 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:25.166441 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:25.166573 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:25.169560 (+ 2987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:25.169563 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:25.169565 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:25.169595 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:25.171415 (+ 1820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:25.172092 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:25.172098 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:25.172102 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:25.172207 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:25.172597 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:25.172600 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:25.172602 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:25.172818 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:25.172944 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:25.173457 (+ 513us) server_negotiation.cc:299] Negotiation successful
0926 07:30:25.173597 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 07:30:25.174600 30653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:25.159465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:39267 (local address 127.22.53.195:49577)
0926 07:30:25.160056 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:25.160102 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:25.161195 (+ 1093us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:25.162096 (+ 901us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:25.162104 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:25.162545 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:25.163264 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:25.163283 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:25.164635 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:25.164639 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:25.165810 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:25.165824 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:25.166639 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:25.166643 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:25.166843 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:25.167527 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:25.167545 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:25.169331 (+ 1786us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:30:25.171557 (+ 2226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:25.171564 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:25.171580 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:25.171956 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:25.172310 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:25.172314 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:25.172316 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:25.172463 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:25.172941 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:30:25.172948 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:30:25.173201 (+ 253us) client_negotiation.cc:769] Sending connection context
0926 07:30:25.173408 (+ 207us) client_negotiation.cc:240] Negotiation successful
0926 07:30:25.173746 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":149,"threads_started":1}
I20260926 07:30:25.176354 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 30629
I20260926 07:30:25.176571 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407783529717-22743-0/minicluster-data/ts-2/wal/instance
I20260926 07:30:25.176908 22743 external_mini_cluster.cc:1428] Setting key ec7c194afb9421b8096c7069d850d25c
I20260926 07:30:25.177300 30760 heartbeater.cc:347] Connected to a master server at 127.22.53.254:39267
I20260926 07:30:25.177510 30760 heartbeater.cc:464] Registering TS with master...
I20260926 07:30:25.178284 30760 heartbeater.cc:511] Master 127.22.53.254:39267 requested a full tablet report, sending...
I20260926 07:30:25.179658 30276 ts_manager.cc:194] Registered new tserver with Master: 6d6bb899bd634ecd9fb3952a365d22ad (127.22.53.195:34909)
I20260926 07:30:25.180346 30276 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.53.195@KRBTEST.COM'} at 127.22.53.195:49577
I20260926 07:30:25.187811 30326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:25.179119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.195:39397 (local address 127.22.53.254:39267)
0926 07:30:25.179249 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:25.179254 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:25.179569 (+ 315us) server_negotiation.cc:407] Connection header received
0926 07:30:25.179693 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:25.179696 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:25.179754 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:25.179876 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:25.180519 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:25.181274 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:25.182534 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:25.182760 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:25.182856 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:25.184346 (+ 1490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:25.184350 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:25.184352 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:25.184381 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:25.185854 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:25.186419 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:25.186422 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:25.186424 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:25.186509 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:25.186872 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:25.186874 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:25.186876 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:25.187095 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:25.187192 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:25.187568 (+ 376us) server_negotiation.cc:299] Negotiation successful
0926 07:30:25.187683 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 07:30:25.187829 30766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:25.179041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:39267 (local address 127.22.53.195:39397)
0926 07:30:25.179450 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:25.179467 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:25.179596 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:25.179928 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:25.179931 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:25.180209 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:25.180343 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:25.180349 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:25.181455 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:25.181460 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:25.182359 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:25.182369 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:25.182900 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:25.182904 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:25.183021 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:25.183682 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:25.183696 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:25.184106 (+ 410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:30:25.185989 (+ 1883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:25.185992 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:25.185994 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:25.186227 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:25.186638 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:25.186641 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:25.186643 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:25.186741 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:25.187218 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:30:25.187220 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:30:25.187389 (+ 169us) client_negotiation.cc:769] Sending connection context
0926 07:30:25.187495 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 07:30:25.187642 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":148,"threads_started":1}
I20260926 07:30:25.191784 22743 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 07:30:25 dist-test-slave-wdjc krb5kdc[30232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407825, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 07:30:25 dist-test-slave-wdjc krb5kdc[30232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.22.53.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:30:25 dist-test-slave-wdjc krb5kdc[30232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.22.53.254@KRBTEST.COM, Server not found in Kerberos database
W20260926 07:30:25.219770 30326 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:30:25.211627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40506 (local address 127.22.53.254:39267)
0926 07:30:25.211770 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:25.211774 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:25.211787 (+ 13us) server_negotiation.cc:407] Connection header received
0926 07:30:25.211832 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:25.211834 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:25.211876 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:25.212008 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:25.212999 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:25.213590 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:25.214750 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:25.215134 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:25.215244 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:25.219574 (+ 4330us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40506: BlockingRecv error: recv got EOF from 127.0.0.1:40506 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 26 07:30:25 dist-test-slave-wdjc krb5kdc[30232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407825, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM
I20260926 07:30:25.231523 30326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:25.221770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40510 (local address 127.22.53.254:39267)
0926 07:30:25.221941 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:25.221946 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:25.222073 (+ 127us) server_negotiation.cc:407] Connection header received
0926 07:30:25.222275 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:25.222278 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:25.222327 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:25.222421 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:25.223284 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:25.223825 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:25.224835 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:25.225074 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:25.225195 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:25.228004 (+ 2809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:25.228008 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:25.228010 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:25.228044 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:25.229642 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:25.230178 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:25.230182 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:25.230184 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:25.230281 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:25.230591 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:25.230595 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:25.230596 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:25.230808 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:25.230958 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:25.231202 (+ 244us) server_negotiation.cc:299] Negotiation successful
0926 07:30:25.231342 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 07:30:25.235008 30276 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40510:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:30:25.238464 30276 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:30:25.255044 30784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:25.248418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.193:34137 (local address 127.0.0.1:42458)
0926 07:30:25.249325 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:25.249342 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:25.249515 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:25.250247 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:25.250250 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:25.250281 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:30:25.250401 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:25.250408 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:25.251661 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:25.251664 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:25.253192 (+ 1528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:25.253203 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:25.254556 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:25.254559 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:25.254627 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:25.254645 (+ 18us) client_negotiation.cc:769] Sending connection context
0926 07:30:25.254744 (+ 99us) client_negotiation.cc:240] Negotiation successful
0926 07:30:25.254886 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":811,"mutex_wait_us":75,"thread_start_us":150,"threads_started":1}
I20260926 07:30:25.255044 30785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:25.248751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42458 (local address 127.22.53.193:34137)
0926 07:30:25.249140 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:25.249147 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:25.249434 (+ 287us) server_negotiation.cc:407] Connection header received
0926 07:30:25.249691 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:25.249697 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:25.249897 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:25.250049 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:30:25.250578 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:25.251503 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:25.253390 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:25.254171 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:25.254289 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:25.254448 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:25.254783 (+ 335us) server_negotiation.cc:299] Negotiation successful
0926 07:30:25.254892 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":94,"threads_started":1}
I20260926 07:30:25.255067 30783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:25.248388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59978 (local address 127.22.53.195:34909)
0926 07:30:25.248864 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:25.248875 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:25.249279 (+ 404us) server_negotiation.cc:407] Connection header received
0926 07:30:25.249606 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:25.249612 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:25.249802 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:25.249910 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:30:25.250482 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:25.251639 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:25.253301 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:25.254169 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:25.254288 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:25.254476 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:25.254712 (+ 236us) server_negotiation.cc:299] Negotiation successful
0926 07:30:25.254840 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":145,"threads_started":1}
I20260926 07:30:25.255533 30786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:25.249436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.194:38789 (local address 127.0.0.1:36126)
0926 07:30:25.250293 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:25.250308 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:25.250401 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:25.250847 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:25.250851 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:25.250872 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:30:25.250970 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:25.250977 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:25.252862 (+ 1885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:25.252866 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:25.253799 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:25.253809 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:25.255035 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:25.255039 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:25.255150 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:25.255184 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 07:30:25.255266 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 07:30:25.255376 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":798,"thread_start_us":107,"threads_started":1}
I20260926 07:30:25.256022 30782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:25.248266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.195:34909 (local address 127.0.0.1:59978)
0926 07:30:25.249124 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:25.249179 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:25.249397 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:25.250164 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:25.250173 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:25.250208 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:30:25.250318 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:25.250322 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:25.251787 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:25.251790 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:25.253179 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:25.253188 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:25.254322 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:25.254327 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:25.254442 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:25.254492 (+ 50us) client_negotiation.cc:769] Sending connection context
0926 07:30:25.255761 (+ 1269us) client_negotiation.cc:240] Negotiation successful
0926 07:30:25.255850 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":695,"mutex_wait_us":124,"thread_start_us":102,"threads_started":1}
I20260926 07:30:25.256299 30787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:25.249783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36126 (local address 127.22.53.194:38789)
0926 07:30:25.250144 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:25.250155 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:25.250408 (+ 253us) server_negotiation.cc:407] Connection header received
0926 07:30:25.250504 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:25.250509 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:25.250691 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:25.250842 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:30:25.251096 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:25.252738 (+ 1642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:25.253937 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:25.254678 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:25.254801 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:25.254964 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:25.255646 (+ 682us) server_negotiation.cc:299] Negotiation successful
0926 07:30:25.256121 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":163,"threads_started":1}
I20260926 07:30:25.258839 30414 tablet_service.cc:1467] Processing CreateTablet for tablet 4125dc1cbff94194bc04385c0a091826 (DEFAULT_TABLE table=test-table [id=78bfb20983fb4b4ea10e27a680732b9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:30:25.258976 30694 tablet_service.cc:1467] Processing CreateTablet for tablet 4125dc1cbff94194bc04385c0a091826 (DEFAULT_TABLE table=test-table [id=78bfb20983fb4b4ea10e27a680732b9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:30:25.258968 30554 tablet_service.cc:1467] Processing CreateTablet for tablet 4125dc1cbff94194bc04385c0a091826 (DEFAULT_TABLE table=test-table [id=78bfb20983fb4b4ea10e27a680732b9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:30:25.260094 30554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4125dc1cbff94194bc04385c0a091826. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:30:25.260094 30694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4125dc1cbff94194bc04385c0a091826. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:30:25.260102 30414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4125dc1cbff94194bc04385c0a091826. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:30:25.267575 30788 tablet_bootstrap.cc:492] T 4125dc1cbff94194bc04385c0a091826 P 39b2209b30574a95bf503e074a4f7d29: Bootstrap starting.
I20260926 07:30:25.269380 30789 tablet_bootstrap.cc:492] T 4125dc1cbff94194bc04385c0a091826 P 6d6bb899bd634ecd9fb3952a365d22ad: Bootstrap starting.
I20260926 07:30:25.270319 30788 tablet_bootstrap.cc:654] T 4125dc1cbff94194bc04385c0a091826 P 39b2209b30574a95bf503e074a4f7d29: Neither blocks nor log segments found. Creating new log.
I20260926 07:30:25.271211 30788 log.cc:824] T 4125dc1cbff94194bc04385c0a091826 P 39b2209b30574a95bf503e074a4f7d29: Log is configured to *not* fsync() on all Append() calls
I20260926 07:30:25.271467 30789 tablet_bootstrap.cc:654] T 4125dc1cbff94194bc04385c0a091826 P 6d6bb899bd634ecd9fb3952a365d22ad: Neither blocks nor log segments found. Creating new log.
I20260926 07:30:25.272341 30789 log.cc:824] T 4125dc1cbff94194bc04385c0a091826 P 6d6bb899bd634ecd9fb3952a365d22ad: Log is configured to *not* fsync() on all Append() calls
I20260926 07:30:25.272390 30790 tablet_bootstrap.cc:492] T 4125dc1cbff94194bc04385c0a091826 P d3cd339e18e549a9b1b07935fe4dd0fc: Bootstrap starting.
I20260926 07:30:25.274353 30789 tablet_bootstrap.cc:492] T 4125dc1cbff94194bc04385c0a091826 P 6d6bb899bd634ecd9fb3952a365d22ad: No bootstrap required, opened a new log
I20260926 07:30:25.274350 30790 tablet_bootstrap.cc:654] T 4125dc1cbff94194bc04385c0a091826 P d3cd339e18e549a9b1b07935fe4dd0fc: Neither blocks nor log segments found. Creating new log.
I20260926 07:30:25.274603 30789 ts_tablet_manager.cc:1397] T 4125dc1cbff94194bc04385c0a091826 P 6d6bb899bd634ecd9fb3952a365d22ad: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 07:30:25.275315 30790 log.cc:824] T 4125dc1cbff94194bc04385c0a091826 P d3cd339e18e549a9b1b07935fe4dd0fc: Log is configured to *not* fsync() on all Append() calls
I20260926 07:30:25.276031 30788 tablet_bootstrap.cc:492] T 4125dc1cbff94194bc04385c0a091826 P 39b2209b30574a95bf503e074a4f7d29: No bootstrap required, opened a new log
I20260926 07:30:25.276405 30788 ts_tablet_manager.cc:1397] T 4125dc1cbff94194bc04385c0a091826 P 39b2209b30574a95bf503e074a4f7d29: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260926 07:30:25.278038 30790 tablet_bootstrap.cc:492] T 4125dc1cbff94194bc04385c0a091826 P d3cd339e18e549a9b1b07935fe4dd0fc: No bootstrap required, opened a new log
I20260926 07:30:25.278302 30790 ts_tablet_manager.cc:1397] T 4125dc1cbff94194bc04385c0a091826 P d3cd339e18e549a9b1b07935fe4dd0fc: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 07:30:25.278333 30789 raft_consensus.cc:348] T 4125dc1cbff94194bc04385c0a091826 P 6d6bb899bd634ecd9fb3952a365d22ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d6bb899bd634ecd9fb3952a365d22ad" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 34909 } } peers { permanent_uuid: "39b2209b30574a95bf503e074a4f7d29" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 34137 } } peers { permanent_uuid: "d3cd339e18e549a9b1b07935fe4dd0fc" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 38789 } }
I20260926 07:30:25.278687 30789 raft_consensus.cc:374] T 4125dc1cbff94194bc04385c0a091826 P 6d6bb899bd634ecd9fb3952a365d22ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:30:25.278745 30789 raft_consensus.cc:729] T 4125dc1cbff94194bc04385c0a091826 P 6d6bb899bd634ecd9fb3952a365d22ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d6bb899bd634ecd9fb3952a365d22ad, State: Initialized, Role: FOLLOWER
I20260926 07:30:25.279286 30789 consensus_queue.cc:260] T 4125dc1cbff94194bc04385c0a091826 P 6d6bb899bd634ecd9fb3952a365d22ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d6bb899bd634ecd9fb3952a365d22ad" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 34909 } } peers { permanent_uuid: "39b2209b30574a95bf503e074a4f7d29" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 34137 } } peers { permanent_uuid: "d3cd339e18e549a9b1b07935fe4dd0fc" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 38789 } }
I20260926 07:30:25.280398 30789 ts_tablet_manager.cc:1428] T 4125dc1cbff94194bc04385c0a091826 P 6d6bb899bd634ecd9fb3952a365d22ad: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 07:30:25.280855 30760 heartbeater.cc:503] Master 127.22.53.254:39267 was elected leader, sending a full tablet report...
I20260926 07:30:25.281656 30788 raft_consensus.cc:348] T 4125dc1cbff94194bc04385c0a091826 P 39b2209b30574a95bf503e074a4f7d29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d6bb899bd634ecd9fb3952a365d22ad" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 34909 } } peers { permanent_uuid: "39b2209b30574a95bf503e074a4f7d29" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 34137 } } peers { permanent_uuid: "d3cd339e18e549a9b1b07935fe4dd0fc" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 38789 } }
I20260926 07:30:25.282014 30788 raft_consensus.cc:374] T 4125dc1cbff94194bc04385c0a091826 P 39b2209b30574a95bf503e074a4f7d29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:30:25.282083 30788 raft_consensus.cc:729] T 4125dc1cbff94194bc04385c0a091826 P 39b2209b30574a95bf503e074a4f7d29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39b2209b30574a95bf503e074a4f7d29, State: Initialized, Role: FOLLOWER
I20260926 07:30:25.282644 30788 consensus_queue.cc:260] T 4125dc1cbff94194bc04385c0a091826 P 39b2209b30574a95bf503e074a4f7d29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d6bb899bd634ecd9fb3952a365d22ad" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 34909 } } peers { permanent_uuid: "39b2209b30574a95bf503e074a4f7d29" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 34137 } } peers { permanent_uuid: "d3cd339e18e549a9b1b07935fe4dd0fc" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 38789 } }
I20260926 07:30:25.282562 30790 raft_consensus.cc:348] T 4125dc1cbff94194bc04385c0a091826 P d3cd339e18e549a9b1b07935fe4dd0fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d6bb899bd634ecd9fb3952a365d22ad" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 34909 } } peers { permanent_uuid: "39b2209b30574a95bf503e074a4f7d29" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 34137 } } peers { permanent_uuid: "d3cd339e18e549a9b1b07935fe4dd0fc" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 38789 } }
I20260926 07:30:25.282845 30790 raft_consensus.cc:374] T 4125dc1cbff94194bc04385c0a091826 P d3cd339e18e549a9b1b07935fe4dd0fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:30:25.282924 30790 raft_consensus.cc:729] T 4125dc1cbff94194bc04385c0a091826 P d3cd339e18e549a9b1b07935fe4dd0fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cd339e18e549a9b1b07935fe4dd0fc, State: Initialized, Role: FOLLOWER
I20260926 07:30:25.283439 30790 consensus_queue.cc:260] T 4125dc1cbff94194bc04385c0a091826 P d3cd339e18e549a9b1b07935fe4dd0fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d6bb899bd634ecd9fb3952a365d22ad" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 34909 } } peers { permanent_uuid: "39b2209b30574a95bf503e074a4f7d29" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 34137 } } peers { permanent_uuid: "d3cd339e18e549a9b1b07935fe4dd0fc" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 38789 } }
I20260926 07:30:25.283862 30788 ts_tablet_manager.cc:1428] T 4125dc1cbff94194bc04385c0a091826 P 39b2209b30574a95bf503e074a4f7d29: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260926 07:30:25.284260 30620 heartbeater.cc:503] Master 127.22.53.254:39267 was elected leader, sending a full tablet report...
I20260926 07:30:25.284260 30480 heartbeater.cc:503] Master 127.22.53.254:39267 was elected leader, sending a full tablet report...
I20260926 07:30:25.284560 30790 ts_tablet_manager.cc:1428] T 4125dc1cbff94194bc04385c0a091826 P d3cd339e18e549a9b1b07935fe4dd0fc: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260926 07:30:25.368031 30794 raft_consensus.cc:482] T 4125dc1cbff94194bc04385c0a091826 P 6d6bb899bd634ecd9fb3952a365d22ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:30:25.368270 30794 raft_consensus.cc:504] T 4125dc1cbff94194bc04385c0a091826 P 6d6bb899bd634ecd9fb3952a365d22ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d6bb899bd634ecd9fb3952a365d22ad" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 34909 } } peers { permanent_uuid: "39b2209b30574a95bf503e074a4f7d29" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 34137 } } peers { permanent_uuid: "d3cd339e18e549a9b1b07935fe4dd0fc" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 38789 } }
I20260926 07:30:25.369460 30794 leader_election.cc:288] T 4125dc1cbff94194bc04385c0a091826 P 6d6bb899bd634ecd9fb3952a365d22ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39b2209b30574a95bf503e074a4f7d29 (127.22.53.193:34137), d3cd339e18e549a9b1b07935fe4dd0fc (127.22.53.194:38789)
I20260926 07:30:25.374015 30653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:25.369790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.194:38789 (local address 127.22.53.195:46289)
0926 07:30:25.369918 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:25.369935 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:25.370054 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:25.370351 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:25.370356 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:25.370383 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:30:25.370530 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:25.370537 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:25.371398 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:25.371400 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:25.372491 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:25.372505 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:25.373472 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:25.373475 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:25.373603 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:25.373631 (+ 28us) client_negotiation.cc:769] Sending connection context
0926 07:30:25.373719 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 07:30:25.373804 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260926 07:30:25.374020 30787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:25.369899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.195:46289 (local address 127.22.53.194:38789)
0926 07:30:25.370033 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:25.370039 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:25.370060 (+ 21us) server_negotiation.cc:407] Connection header received
0926 07:30:25.370165 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:25.370171 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:25.370230 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:25.370314 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:30:25.370660 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:25.371244 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:25.372766 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:25.373346 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:25.373447 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:25.373492 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:25.373780 (+ 288us) server_negotiation.cc:299] Negotiation successful
0926 07:30:25.373859 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 07:30:25.374524 30785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:25.370010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.195:35917 (local address 127.22.53.193:34137)
0926 07:30:25.370138 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:25.370156 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:25.370457 (+ 301us) server_negotiation.cc:407] Connection header received
0926 07:30:25.370522 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:25.370524 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:25.370563 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:25.370664 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:30:25.371276 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:25.371829 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:25.373219 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:25.373838 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:25.373955 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:25.374003 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:25.374312 (+ 309us) server_negotiation.cc:299] Negotiation successful
0926 07:30:25.374394 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 07:30:25.374536 30797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:25.369790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.193:34137 (local address 127.22.53.195:35917)
0926 07:30:25.370347 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:25.370361 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:25.370446 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:25.370689 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:25.370691 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:25.370706 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:30:25.371147 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:25.371152 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:25.371970 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:25.371972 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:25.373090 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:25.373101 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:25.373978 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:25.373982 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:25.374102 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:25.374130 (+ 28us) client_negotiation.cc:769] Sending connection context
0926 07:30:25.374269 (+ 139us) client_negotiation.cc:240] Negotiation successful
0926 07:30:25.374352 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"mutex_wait_us":140,"thread_start_us":147,"threads_started":1}
I20260926 07:30:25.374964 30574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4125dc1cbff94194bc04385c0a091826" candidate_uuid: "6d6bb899bd634ecd9fb3952a365d22ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cd339e18e549a9b1b07935fe4dd0fc" is_pre_election: true
I20260926 07:30:25.375344 30574 raft_consensus.cc:2454] T 4125dc1cbff94194bc04385c0a091826 P d3cd339e18e549a9b1b07935fe4dd0fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d6bb899bd634ecd9fb3952a365d22ad in term 0.
I20260926 07:30:25.375350 30434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4125dc1cbff94194bc04385c0a091826" candidate_uuid: "6d6bb899bd634ecd9fb3952a365d22ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39b2209b30574a95bf503e074a4f7d29" is_pre_election: true
I20260926 07:30:25.375768 30434 raft_consensus.cc:2454] T 4125dc1cbff94194bc04385c0a091826 P 39b2209b30574a95bf503e074a4f7d29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d6bb899bd634ecd9fb3952a365d22ad in term 0.
I20260926 07:30:25.375914 30646 leader_election.cc:302] T 4125dc1cbff94194bc04385c0a091826 P 6d6bb899bd634ecd9fb3952a365d22ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d6bb899bd634ecd9fb3952a365d22ad, d3cd339e18e549a9b1b07935fe4dd0fc; no voters:
I20260926 07:30:25.376228 30794 raft_consensus.cc:2781] T 4125dc1cbff94194bc04385c0a091826 P 6d6bb899bd634ecd9fb3952a365d22ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:30:25.376338 30794 raft_consensus.cc:482] T 4125dc1cbff94194bc04385c0a091826 P 6d6bb899bd634ecd9fb3952a365d22ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:30:25.376420 30794 raft_consensus.cc:3037] T 4125dc1cbff94194bc04385c0a091826 P 6d6bb899bd634ecd9fb3952a365d22ad [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:30:25.377655 30794 raft_consensus.cc:504] T 4125dc1cbff94194bc04385c0a091826 P 6d6bb899bd634ecd9fb3952a365d22ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d6bb899bd634ecd9fb3952a365d22ad" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 34909 } } peers { permanent_uuid: "39b2209b30574a95bf503e074a4f7d29" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 34137 } } peers { permanent_uuid: "d3cd339e18e549a9b1b07935fe4dd0fc" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 38789 } }
I20260926 07:30:25.378104 30794 leader_election.cc:288] T 4125dc1cbff94194bc04385c0a091826 P 6d6bb899bd634ecd9fb3952a365d22ad [CANDIDATE]: Term 1 election: Requested vote from peers 39b2209b30574a95bf503e074a4f7d29 (127.22.53.193:34137), d3cd339e18e549a9b1b07935fe4dd0fc (127.22.53.194:38789)
I20260926 07:30:25.378499 30434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4125dc1cbff94194bc04385c0a091826" candidate_uuid: "6d6bb899bd634ecd9fb3952a365d22ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39b2209b30574a95bf503e074a4f7d29"
I20260926 07:30:25.378558 30574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4125dc1cbff94194bc04385c0a091826" candidate_uuid: "6d6bb899bd634ecd9fb3952a365d22ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cd339e18e549a9b1b07935fe4dd0fc"
I20260926 07:30:25.378665 30434 raft_consensus.cc:3037] T 4125dc1cbff94194bc04385c0a091826 P 39b2209b30574a95bf503e074a4f7d29 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:30:25.378696 30574 raft_consensus.cc:3037] T 4125dc1cbff94194bc04385c0a091826 P d3cd339e18e549a9b1b07935fe4dd0fc [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:30:25.379848 30434 raft_consensus.cc:2454] T 4125dc1cbff94194bc04385c0a091826 P 39b2209b30574a95bf503e074a4f7d29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d6bb899bd634ecd9fb3952a365d22ad in term 1.
I20260926 07:30:25.379863 30574 raft_consensus.cc:2454] T 4125dc1cbff94194bc04385c0a091826 P d3cd339e18e549a9b1b07935fe4dd0fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d6bb899bd634ecd9fb3952a365d22ad in term 1.
I20260926 07:30:25.380223 30646 leader_election.cc:302] T 4125dc1cbff94194bc04385c0a091826 P 6d6bb899bd634ecd9fb3952a365d22ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d6bb899bd634ecd9fb3952a365d22ad, d3cd339e18e549a9b1b07935fe4dd0fc; no voters:
I20260926 07:30:25.380465 30794 raft_consensus.cc:2781] T 4125dc1cbff94194bc04385c0a091826 P 6d6bb899bd634ecd9fb3952a365d22ad [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:30:25.380784 30794 raft_consensus.cc:686] T 4125dc1cbff94194bc04385c0a091826 P 6d6bb899bd634ecd9fb3952a365d22ad [term 1 LEADER]: Becoming Leader. State: Replica: 6d6bb899bd634ecd9fb3952a365d22ad, State: Running, Role: LEADER
I20260926 07:30:25.381186 30794 consensus_queue.cc:237] T 4125dc1cbff94194bc04385c0a091826 P 6d6bb899bd634ecd9fb3952a365d22ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d6bb899bd634ecd9fb3952a365d22ad" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 34909 } } peers { permanent_uuid: "39b2209b30574a95bf503e074a4f7d29" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 34137 } } peers { permanent_uuid: "d3cd339e18e549a9b1b07935fe4dd0fc" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 38789 } }
I20260926 07:30:25.384860 30277 catalog_manager.cc:5495] T 4125dc1cbff94194bc04385c0a091826 P 6d6bb899bd634ecd9fb3952a365d22ad reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d6bb899bd634ecd9fb3952a365d22ad (127.22.53.195). New cstate: current_term: 1 leader_uuid: "6d6bb899bd634ecd9fb3952a365d22ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d6bb899bd634ecd9fb3952a365d22ad" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 34909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39b2209b30574a95bf503e074a4f7d29" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 34137 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3cd339e18e549a9b1b07935fe4dd0fc" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 38789 } health_report { overall_health: UNKNOWN } } }
I20260926 07:30:25.411787 30783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:25.407984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59986 (local address 127.22.53.195:34909)
0926 07:30:25.408127 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:25.408130 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:25.408145 (+ 15us) server_negotiation.cc:407] Connection header received
0926 07:30:25.408243 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:25.408245 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:25.408304 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:25.408406 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:30:25.408768 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:25.409284 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:25.410178 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:25.410414 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:25.410532 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:25.410857 (+ 325us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:30:25.411347 (+ 490us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:30:25.411486 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:25.411554 (+ 68us) server_negotiation.cc:299] Negotiation successful
0926 07:30:25.411622 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 07:30:25.418265 30574 raft_consensus.cc:1261] T 4125dc1cbff94194bc04385c0a091826 P d3cd339e18e549a9b1b07935fe4dd0fc [term 1 FOLLOWER]: Refusing update from remote peer 6d6bb899bd634ecd9fb3952a365d22ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:30:25.418265 30434 raft_consensus.cc:1261] T 4125dc1cbff94194bc04385c0a091826 P 39b2209b30574a95bf503e074a4f7d29 [term 1 FOLLOWER]: Refusing update from remote peer 6d6bb899bd634ecd9fb3952a365d22ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:30:25.419113 30794 consensus_queue.cc:1035] T 4125dc1cbff94194bc04385c0a091826 P 6d6bb899bd634ecd9fb3952a365d22ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "39b2209b30574a95bf503e074a4f7d29" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 34137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:30:25.419335 30798 consensus_queue.cc:1035] T 4125dc1cbff94194bc04385c0a091826 P 6d6bb899bd634ecd9fb3952a365d22ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3cd339e18e549a9b1b07935fe4dd0fc" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 38789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 07:30:25.422371 30761 tablet.cc:2369] T 4125dc1cbff94194bc04385c0a091826 P 6d6bb899bd634ecd9fb3952a365d22ad: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:30:25.428885 30802 mvcc.cc:205] Tried to move back new op lower bound from 7333510452905295872 to 7333510452764377088. Current Snapshot: MvccSnapshot[applied={T|T < 7333510452905295872}]
I20260926 07:30:25.429203 30801 mvcc.cc:205] Tried to move back new op lower bound from 7333510452905295872 to 7333510452764377088. Current Snapshot: MvccSnapshot[applied={T|T < 7333510452905295872}]
I20260926 07:30:25.438141 30277 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:30:25.438356 30277 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:30:25.441421 30277 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 945929f517c847deaee9a27a48db5aa9: Sending DeleteTablet for 3 replicas of tablet 4125dc1cbff94194bc04385c0a091826
I20260926 07:30:25.442382 30414 tablet_service.cc:1514] Processing DeleteTablet for tablet 4125dc1cbff94194bc04385c0a091826 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:30:25 UTC) from {username='oryx', principal='oryx/127.22.53.254@KRBTEST.COM'} at 127.0.0.1:42458
I20260926 07:30:25.442376 30694 tablet_service.cc:1514] Processing DeleteTablet for tablet 4125dc1cbff94194bc04385c0a091826 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:30:25 UTC) from {username='oryx', principal='oryx/127.22.53.254@KRBTEST.COM'} at 127.0.0.1:59978
I20260926 07:30:25.442648 30554 tablet_service.cc:1514] Processing DeleteTablet for tablet 4125dc1cbff94194bc04385c0a091826 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:30:25 UTC) from {username='oryx', principal='oryx/127.22.53.254@KRBTEST.COM'} at 127.0.0.1:36126
I20260926 07:30:25.443066 30811 tablet_replica.cc:330] T 4125dc1cbff94194bc04385c0a091826 P 39b2209b30574a95bf503e074a4f7d29: stopping tablet replica
I20260926 07:30:25.443213 30813 tablet_replica.cc:330] T 4125dc1cbff94194bc04385c0a091826 P d3cd339e18e549a9b1b07935fe4dd0fc: stopping tablet replica
I20260926 07:30:25.443621 30813 raft_consensus.cc:2229] T 4125dc1cbff94194bc04385c0a091826 P d3cd339e18e549a9b1b07935fe4dd0fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:30:25.443944 30811 raft_consensus.cc:2229] T 4125dc1cbff94194bc04385c0a091826 P 39b2209b30574a95bf503e074a4f7d29 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:30:25.444187 30813 raft_consensus.cc:2258] T 4125dc1cbff94194bc04385c0a091826 P d3cd339e18e549a9b1b07935fe4dd0fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:30:25.444280 30811 raft_consensus.cc:2258] T 4125dc1cbff94194bc04385c0a091826 P 39b2209b30574a95bf503e074a4f7d29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:30:25.444159 30277 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40510:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 07:30:25.444594 30812 tablet_replica.cc:330] T 4125dc1cbff94194bc04385c0a091826 P 6d6bb899bd634ecd9fb3952a365d22ad: stopping tablet replica
I20260926 07:30:25.444903 30812 raft_consensus.cc:2229] T 4125dc1cbff94194bc04385c0a091826 P 6d6bb899bd634ecd9fb3952a365d22ad [term 1 LEADER]: Raft consensus shutting down.
W20260926 07:30:25.445010 30277 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:30:25.445360 30812 raft_consensus.cc:2258] T 4125dc1cbff94194bc04385c0a091826 P 6d6bb899bd634ecd9fb3952a365d22ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:30:25.446050 30813 ts_tablet_manager.cc:1905] T 4125dc1cbff94194bc04385c0a091826 P d3cd339e18e549a9b1b07935fe4dd0fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:30:25.446153 30811 ts_tablet_manager.cc:1905] T 4125dc1cbff94194bc04385c0a091826 P 39b2209b30574a95bf503e074a4f7d29: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:30:25.449012 30813 ts_tablet_manager.cc:1918] T 4125dc1cbff94194bc04385c0a091826 P d3cd339e18e549a9b1b07935fe4dd0fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:30:25.449128 30813 log.cc:1196] T 4125dc1cbff94194bc04385c0a091826 P d3cd339e18e549a9b1b07935fe4dd0fc: Deleting WAL directory at /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407783529717-22743-0/minicluster-data/ts-1/wal/wals/4125dc1cbff94194bc04385c0a091826
I20260926 07:30:25.449187 30811 ts_tablet_manager.cc:1918] T 4125dc1cbff94194bc04385c0a091826 P 39b2209b30574a95bf503e074a4f7d29: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:30:25.449286 30811 log.cc:1196] T 4125dc1cbff94194bc04385c0a091826 P 39b2209b30574a95bf503e074a4f7d29: Deleting WAL directory at /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407783529717-22743-0/minicluster-data/ts-0/wal/wals/4125dc1cbff94194bc04385c0a091826
I20260926 07:30:25.449461 30813 ts_tablet_manager.cc:1939] T 4125dc1cbff94194bc04385c0a091826 P d3cd339e18e549a9b1b07935fe4dd0fc: Deleting consensus metadata
I20260926 07:30:25.449600 30811 ts_tablet_manager.cc:1939] T 4125dc1cbff94194bc04385c0a091826 P 39b2209b30574a95bf503e074a4f7d29: Deleting consensus metadata
I20260926 07:30:25.450930 30414 tablet_service.cc:1467] Processing CreateTablet for tablet 0545bdb9f17e41ec95c8c2e9df8561be (DEFAULT_TABLE table=test-table [id=864b71dfdf7c40d9bc69f2ddc318a8a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:30:25.451005 30263 catalog_manager.cc:4841] TS 39b2209b30574a95bf503e074a4f7d29 (127.22.53.193:34137): tablet 4125dc1cbff94194bc04385c0a091826 (table test-table [id=78bfb20983fb4b4ea10e27a680732b9f]) successfully deleted
I20260926 07:30:25.451218 30414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0545bdb9f17e41ec95c8c2e9df8561be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:30:25.451164 30554 tablet_service.cc:1467] Processing CreateTablet for tablet 0545bdb9f17e41ec95c8c2e9df8561be (DEFAULT_TABLE table=test-table [id=864b71dfdf7c40d9bc69f2ddc318a8a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:30:25.451397 30554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0545bdb9f17e41ec95c8c2e9df8561be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:30:25.453719 30788 tablet_bootstrap.cc:492] T 0545bdb9f17e41ec95c8c2e9df8561be P 39b2209b30574a95bf503e074a4f7d29: Bootstrap starting.
I20260926 07:30:25.453771 30790 tablet_bootstrap.cc:492] T 0545bdb9f17e41ec95c8c2e9df8561be P d3cd339e18e549a9b1b07935fe4dd0fc: Bootstrap starting.
I20260926 07:30:25.454607 30790 tablet_bootstrap.cc:654] T 0545bdb9f17e41ec95c8c2e9df8561be P d3cd339e18e549a9b1b07935fe4dd0fc: Neither blocks nor log segments found. Creating new log.
I20260926 07:30:25.454763 30788 tablet_bootstrap.cc:654] T 0545bdb9f17e41ec95c8c2e9df8561be P 39b2209b30574a95bf503e074a4f7d29: Neither blocks nor log segments found. Creating new log.
I20260926 07:30:25.456241 30790 tablet_bootstrap.cc:492] T 0545bdb9f17e41ec95c8c2e9df8561be P d3cd339e18e549a9b1b07935fe4dd0fc: No bootstrap required, opened a new log
I20260926 07:30:25.456210 30694 tablet_service.cc:1467] Processing CreateTablet for tablet 0545bdb9f17e41ec95c8c2e9df8561be (DEFAULT_TABLE table=test-table [id=864b71dfdf7c40d9bc69f2ddc318a8a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:30:25.456322 30788 tablet_bootstrap.cc:492] T 0545bdb9f17e41ec95c8c2e9df8561be P 39b2209b30574a95bf503e074a4f7d29: No bootstrap required, opened a new log
I20260926 07:30:25.456332 30790 ts_tablet_manager.cc:1397] T 0545bdb9f17e41ec95c8c2e9df8561be P d3cd339e18e549a9b1b07935fe4dd0fc: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 07:30:25.456404 30788 ts_tablet_manager.cc:1397] T 0545bdb9f17e41ec95c8c2e9df8561be P 39b2209b30574a95bf503e074a4f7d29: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 07:30:25.456578 30694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0545bdb9f17e41ec95c8c2e9df8561be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:30:25.456885 30790 raft_consensus.cc:348] T 0545bdb9f17e41ec95c8c2e9df8561be P d3cd339e18e549a9b1b07935fe4dd0fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39b2209b30574a95bf503e074a4f7d29" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 34137 } } peers { permanent_uuid: "d3cd339e18e549a9b1b07935fe4dd0fc" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 38789 } } peers { permanent_uuid: "6d6bb899bd634ecd9fb3952a365d22ad" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 34909 } }
I20260926 07:30:25.456912 30788 raft_consensus.cc:348] T 0545bdb9f17e41ec95c8c2e9df8561be P 39b2209b30574a95bf503e074a4f7d29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39b2209b30574a95bf503e074a4f7d29" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 34137 } } peers { permanent_uuid: "d3cd339e18e549a9b1b07935fe4dd0fc" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 38789 } } peers { permanent_uuid: "6d6bb899bd634ecd9fb3952a365d22ad" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 34909 } }
I20260926 07:30:25.457010 30790 raft_consensus.cc:374] T 0545bdb9f17e41ec95c8c2e9df8561be P d3cd339e18e549a9b1b07935fe4dd0fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:30:25.457023 30788 raft_consensus.cc:374] T 0545bdb9f17e41ec95c8c2e9df8561be P 39b2209b30574a95bf503e074a4f7d29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:30:25.457041 30790 raft_consensus.cc:729] T 0545bdb9f17e41ec95c8c2e9df8561be P d3cd339e18e549a9b1b07935fe4dd0fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cd339e18e549a9b1b07935fe4dd0fc, State: Initialized, Role: FOLLOWER
I20260926 07:30:25.457052 30788 raft_consensus.cc:729] T 0545bdb9f17e41ec95c8c2e9df8561be P 39b2209b30574a95bf503e074a4f7d29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39b2209b30574a95bf503e074a4f7d29, State: Initialized, Role: FOLLOWER
I20260926 07:30:25.457165 30790 consensus_queue.cc:260] T 0545bdb9f17e41ec95c8c2e9df8561be P d3cd339e18e549a9b1b07935fe4dd0fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39b2209b30574a95bf503e074a4f7d29" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 34137 } } peers { permanent_uuid: "d3cd339e18e549a9b1b07935fe4dd0fc" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 38789 } } peers { permanent_uuid: "6d6bb899bd634ecd9fb3952a365d22ad" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 34909 } }
I20260926 07:30:25.457171 30788 consensus_queue.cc:260] T 0545bdb9f17e41ec95c8c2e9df8561be P 39b2209b30574a95bf503e074a4f7d29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39b2209b30574a95bf503e074a4f7d29" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 34137 } } peers { permanent_uuid: "d3cd339e18e549a9b1b07935fe4dd0fc" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 38789 } } peers { permanent_uuid: "6d6bb899bd634ecd9fb3952a365d22ad" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 34909 } }
I20260926 07:30:25.457417 30788 ts_tablet_manager.cc:1428] T 0545bdb9f17e41ec95c8c2e9df8561be P 39b2209b30574a95bf503e074a4f7d29: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 07:30:25.457422 30790 ts_tablet_manager.cc:1428] T 0545bdb9f17e41ec95c8c2e9df8561be P d3cd339e18e549a9b1b07935fe4dd0fc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 07:30:25.458882 30262 catalog_manager.cc:4841] TS d3cd339e18e549a9b1b07935fe4dd0fc (127.22.53.194:38789): tablet 4125dc1cbff94194bc04385c0a091826 (table test-table [id=78bfb20983fb4b4ea10e27a680732b9f]) successfully deleted
I20260926 07:30:25.459373 30789 tablet_bootstrap.cc:492] T 0545bdb9f17e41ec95c8c2e9df8561be P 6d6bb899bd634ecd9fb3952a365d22ad: Bootstrap starting.
I20260926 07:30:25.460487 30789 tablet_bootstrap.cc:654] T 0545bdb9f17e41ec95c8c2e9df8561be P 6d6bb899bd634ecd9fb3952a365d22ad: Neither blocks nor log segments found. Creating new log.
I20260926 07:30:25.461978 30789 tablet_bootstrap.cc:492] T 0545bdb9f17e41ec95c8c2e9df8561be P 6d6bb899bd634ecd9fb3952a365d22ad: No bootstrap required, opened a new log
I20260926 07:30:25.462080 30789 ts_tablet_manager.cc:1397] T 0545bdb9f17e41ec95c8c2e9df8561be P 6d6bb899bd634ecd9fb3952a365d22ad: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260926 07:30:25.447580 30812 ts_tablet_manager.cc:1905] T 4125dc1cbff94194bc04385c0a091826 P 6d6bb899bd634ecd9fb3952a365d22ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:30:25.462607 30789 raft_consensus.cc:348] T 0545bdb9f17e41ec95c8c2e9df8561be P 6d6bb899bd634ecd9fb3952a365d22ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39b2209b30574a95bf503e074a4f7d29" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 34137 } } peers { permanent_uuid: "d3cd339e18e549a9b1b07935fe4dd0fc" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 38789 } } peers { permanent_uuid: "6d6bb899bd634ecd9fb3952a365d22ad" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 34909 } }
I20260926 07:30:25.462743 30789 raft_consensus.cc:374] T 0545bdb9f17e41ec95c8c2e9df8561be P 6d6bb899bd634ecd9fb3952a365d22ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:30:25.462787 30789 raft_consensus.cc:729] T 0545bdb9f17e41ec95c8c2e9df8561be P 6d6bb899bd634ecd9fb3952a365d22ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d6bb899bd634ecd9fb3952a365d22ad, State: Initialized, Role: FOLLOWER
I20260926 07:30:25.462925 30789 consensus_queue.cc:260] T 0545bdb9f17e41ec95c8c2e9df8561be P 6d6bb899bd634ecd9fb3952a365d22ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39b2209b30574a95bf503e074a4f7d29" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 34137 } } peers { permanent_uuid: "d3cd339e18e549a9b1b07935fe4dd0fc" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 38789 } } peers { permanent_uuid: "6d6bb899bd634ecd9fb3952a365d22ad" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 34909 } }
I20260926 07:30:25.463189 30789 ts_tablet_manager.cc:1428] T 0545bdb9f17e41ec95c8c2e9df8561be P 6d6bb899bd634ecd9fb3952a365d22ad: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260926 07:30:25.465217 30812 ts_tablet_manager.cc:1918] T 4125dc1cbff94194bc04385c0a091826 P 6d6bb899bd634ecd9fb3952a365d22ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:30:25.465356 30812 log.cc:1196] T 4125dc1cbff94194bc04385c0a091826 P 6d6bb899bd634ecd9fb3952a365d22ad: Deleting WAL directory at /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407783529717-22743-0/minicluster-data/ts-2/wal/wals/4125dc1cbff94194bc04385c0a091826
I20260926 07:30:25.465663 30812 ts_tablet_manager.cc:1939] T 4125dc1cbff94194bc04385c0a091826 P 6d6bb899bd634ecd9fb3952a365d22ad: Deleting consensus metadata
I20260926 07:30:25.466996 30261 catalog_manager.cc:4841] TS 6d6bb899bd634ecd9fb3952a365d22ad (127.22.53.195:34909): tablet 4125dc1cbff94194bc04385c0a091826 (table test-table [id=78bfb20983fb4b4ea10e27a680732b9f]) successfully deleted
I20260926 07:30:25.680401 30796 raft_consensus.cc:482] T 0545bdb9f17e41ec95c8c2e9df8561be P d3cd339e18e549a9b1b07935fe4dd0fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:30:25.680599 30796 raft_consensus.cc:504] T 0545bdb9f17e41ec95c8c2e9df8561be P d3cd339e18e549a9b1b07935fe4dd0fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39b2209b30574a95bf503e074a4f7d29" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 34137 } } peers { permanent_uuid: "d3cd339e18e549a9b1b07935fe4dd0fc" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 38789 } } peers { permanent_uuid: "6d6bb899bd634ecd9fb3952a365d22ad" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 34909 } }
W20260926 07:30:25.681250 30621 tablet.cc:2369] T 0545bdb9f17e41ec95c8c2e9df8561be P d3cd339e18e549a9b1b07935fe4dd0fc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:30:25.682073 30796 leader_election.cc:288] T 0545bdb9f17e41ec95c8c2e9df8561be P d3cd339e18e549a9b1b07935fe4dd0fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39b2209b30574a95bf503e074a4f7d29 (127.22.53.193:34137), 6d6bb899bd634ecd9fb3952a365d22ad (127.22.53.195:34909)
I20260926 07:30:25.687835 30785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:25.682876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.194:60129 (local address 127.22.53.193:34137)
0926 07:30:25.683020 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:25.683023 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:25.683036 (+ 13us) server_negotiation.cc:407] Connection header received
0926 07:30:25.683089 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:25.683091 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:25.683132 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:25.683203 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:30:25.683548 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:25.684370 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:25.686363 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:25.687025 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:25.687181 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:25.687233 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:25.687625 (+ 392us) server_negotiation.cc:299] Negotiation successful
0926 07:30:25.687717 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 07:30:25.687892 30818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:25.682402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.193:34137 (local address 127.22.53.194:60129)
0926 07:30:25.682753 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:25.682767 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:25.682840 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:25.683239 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:25.683241 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:25.683258 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:30:25.683390 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:25.683395 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:25.684510 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:25.684513 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:25.686176 (+ 1663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:25.686191 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:25.687214 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:25.687219 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:25.687446 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:25.687501 (+ 55us) client_negotiation.cc:769] Sending connection context
0926 07:30:25.687588 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 07:30:25.687704 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"mutex_wait_us":221,"spinlock_wait_cycles":12672,"thread_start_us":75,"threads_started":1}
I20260926 07:30:25.687896 30817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:25.682370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.195:34909 (local address 127.22.53.194:54589)
0926 07:30:25.682698 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:25.682716 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:25.682840 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:25.683502 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:25.683505 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:25.683527 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:30:25.683627 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:25.683631 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:25.684634 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:25.684637 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:25.686176 (+ 1539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:25.686191 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:25.687214 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:25.687219 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:25.687449 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:25.687507 (+ 58us) client_negotiation.cc:769] Sending connection context
0926 07:30:25.687587 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 07:30:25.687704 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"mutex_wait_us":252,"thread_start_us":127,"threads_started":1}
I20260926 07:30:25.688503 30434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0545bdb9f17e41ec95c8c2e9df8561be" candidate_uuid: "d3cd339e18e549a9b1b07935fe4dd0fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39b2209b30574a95bf503e074a4f7d29" is_pre_election: true
I20260926 07:30:25.688714 30434 raft_consensus.cc:2454] T 0545bdb9f17e41ec95c8c2e9df8561be P 39b2209b30574a95bf503e074a4f7d29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3cd339e18e549a9b1b07935fe4dd0fc in term 0.
I20260926 07:30:25.688992 30783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:25.682476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.194:54589 (local address 127.22.53.195:34909)
0926 07:30:25.682760 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:25.682765 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:25.682786 (+ 21us) server_negotiation.cc:407] Connection header received
0926 07:30:25.683338 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:25.683341 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:25.683385 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:25.683445 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:30:25.683739 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:25.684476 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:25.686362 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:25.687024 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:25.687173 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:25.687233 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:25.688702 (+ 1469us) server_negotiation.cc:299] Negotiation successful
0926 07:30:25.688814 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175}
I20260926 07:30:25.689254 30507 leader_election.cc:302] T 0545bdb9f17e41ec95c8c2e9df8561be P d3cd339e18e549a9b1b07935fe4dd0fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39b2209b30574a95bf503e074a4f7d29, d3cd339e18e549a9b1b07935fe4dd0fc; no voters:
I20260926 07:30:25.689626 30796 raft_consensus.cc:2781] T 0545bdb9f17e41ec95c8c2e9df8561be P d3cd339e18e549a9b1b07935fe4dd0fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:30:25.689702 30796 raft_consensus.cc:482] T 0545bdb9f17e41ec95c8c2e9df8561be P d3cd339e18e549a9b1b07935fe4dd0fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:30:25.689608 30714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0545bdb9f17e41ec95c8c2e9df8561be" candidate_uuid: "d3cd339e18e549a9b1b07935fe4dd0fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d6bb899bd634ecd9fb3952a365d22ad" is_pre_election: true
I20260926 07:30:25.689741 30796 raft_consensus.cc:3037] T 0545bdb9f17e41ec95c8c2e9df8561be P d3cd339e18e549a9b1b07935fe4dd0fc [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:30:25.689867 30714 raft_consensus.cc:2454] T 0545bdb9f17e41ec95c8c2e9df8561be P 6d6bb899bd634ecd9fb3952a365d22ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3cd339e18e549a9b1b07935fe4dd0fc in term 0.
I20260926 07:30:25.691097 30796 raft_consensus.cc:504] T 0545bdb9f17e41ec95c8c2e9df8561be P d3cd339e18e549a9b1b07935fe4dd0fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39b2209b30574a95bf503e074a4f7d29" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 34137 } } peers { permanent_uuid: "d3cd339e18e549a9b1b07935fe4dd0fc" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 38789 } } peers { permanent_uuid: "6d6bb899bd634ecd9fb3952a365d22ad" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 34909 } }
W20260926 07:30:25.691250 30481 tablet.cc:2369] T 0545bdb9f17e41ec95c8c2e9df8561be P 39b2209b30574a95bf503e074a4f7d29: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:30:25.691766 30796 leader_election.cc:288] T 0545bdb9f17e41ec95c8c2e9df8561be P d3cd339e18e549a9b1b07935fe4dd0fc [CANDIDATE]: Term 1 election: Requested vote from peers 39b2209b30574a95bf503e074a4f7d29 (127.22.53.193:34137), 6d6bb899bd634ecd9fb3952a365d22ad (127.22.53.195:34909)
I20260926 07:30:25.692037 30434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0545bdb9f17e41ec95c8c2e9df8561be" candidate_uuid: "d3cd339e18e549a9b1b07935fe4dd0fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39b2209b30574a95bf503e074a4f7d29"
I20260926 07:30:25.692173 30434 raft_consensus.cc:3037] T 0545bdb9f17e41ec95c8c2e9df8561be P 39b2209b30574a95bf503e074a4f7d29 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:30:25.692188 30714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0545bdb9f17e41ec95c8c2e9df8561be" candidate_uuid: "d3cd339e18e549a9b1b07935fe4dd0fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d6bb899bd634ecd9fb3952a365d22ad"
I20260926 07:30:25.692337 30714 raft_consensus.cc:3037] T 0545bdb9f17e41ec95c8c2e9df8561be P 6d6bb899bd634ecd9fb3952a365d22ad [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:30:25.693486 30714 raft_consensus.cc:2454] T 0545bdb9f17e41ec95c8c2e9df8561be P 6d6bb899bd634ecd9fb3952a365d22ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3cd339e18e549a9b1b07935fe4dd0fc in term 1.
I20260926 07:30:25.693477 30434 raft_consensus.cc:2454] T 0545bdb9f17e41ec95c8c2e9df8561be P 39b2209b30574a95bf503e074a4f7d29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3cd339e18e549a9b1b07935fe4dd0fc in term 1.
I20260926 07:30:25.693857 30507 leader_election.cc:302] T 0545bdb9f17e41ec95c8c2e9df8561be P d3cd339e18e549a9b1b07935fe4dd0fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39b2209b30574a95bf503e074a4f7d29, d3cd339e18e549a9b1b07935fe4dd0fc; no voters:
I20260926 07:30:25.694100 30796 raft_consensus.cc:2781] T 0545bdb9f17e41ec95c8c2e9df8561be P d3cd339e18e549a9b1b07935fe4dd0fc [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:30:25.694332 30796 raft_consensus.cc:686] T 0545bdb9f17e41ec95c8c2e9df8561be P d3cd339e18e549a9b1b07935fe4dd0fc [term 1 LEADER]: Becoming Leader. State: Replica: d3cd339e18e549a9b1b07935fe4dd0fc, State: Running, Role: LEADER
I20260926 07:30:25.694654 30796 consensus_queue.cc:237] T 0545bdb9f17e41ec95c8c2e9df8561be P d3cd339e18e549a9b1b07935fe4dd0fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39b2209b30574a95bf503e074a4f7d29" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 34137 } } peers { permanent_uuid: "d3cd339e18e549a9b1b07935fe4dd0fc" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 38789 } } peers { permanent_uuid: "6d6bb899bd634ecd9fb3952a365d22ad" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 34909 } }
I20260926 07:30:25.697412 30277 catalog_manager.cc:5495] T 0545bdb9f17e41ec95c8c2e9df8561be P d3cd339e18e549a9b1b07935fe4dd0fc reported cstate change: term changed from 0 to 1, leader changed from <none> to d3cd339e18e549a9b1b07935fe4dd0fc (127.22.53.194). New cstate: current_term: 1 leader_uuid: "d3cd339e18e549a9b1b07935fe4dd0fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39b2209b30574a95bf503e074a4f7d29" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 34137 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3cd339e18e549a9b1b07935fe4dd0fc" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 38789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6d6bb899bd634ecd9fb3952a365d22ad" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 34909 } health_report { overall_health: UNKNOWN } } }
Sep 26 07:30:25 dist-test-slave-wdjc krb5kdc[30232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407824, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.53.254@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM
I20260926 07:30:25.732265 30326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:25.722646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40520 (local address 127.22.53.254:39267)
0926 07:30:25.722904 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:25.722910 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:25.723094 (+ 184us) server_negotiation.cc:407] Connection header received
0926 07:30:25.723265 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:25.723268 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:25.723315 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:25.723415 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:25.724133 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:25.724729 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:25.725552 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:25.725772 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:25.725893 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:25.728427 (+ 2534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:25.728433 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:25.728436 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:25.728468 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:25.730045 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:25.730637 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:25.730640 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:25.730642 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:25.730722 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:25.731024 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:25.731027 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:25.731029 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:25.731242 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:25.731406 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:25.731838 (+ 432us) server_negotiation.cc:299] Negotiation successful
0926 07:30:25.732047 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":80}
I20260926 07:30:25.732270 30826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:25.722612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:39267 (local address 127.0.0.1:40520)
0926 07:30:25.722994 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:25.723007 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:25.723161 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:25.723432 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:25.723435 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:25.723894 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:25.723988 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:25.724010 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:25.724857 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:25.724862 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:25.725425 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:25.725433 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:25.725899 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:25.725902 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:25.725975 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:25.726612 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:25.726625 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:25.728248 (+ 1623us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:30:25.730211 (+ 1963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:25.730226 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:25.730233 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:25.730509 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:25.730835 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:25.730838 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:25.730840 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:25.730921 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:25.731487 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:30:25.731504 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:30:25.731643 (+ 139us) client_negotiation.cc:769] Sending connection context
0926 07:30:25.731791 (+ 148us) client_negotiation.cc:240] Negotiation successful
0926 07:30:25.732108 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"mutex_wait_us":129,"thread_start_us":127,"threads_started":1}
I20260926 07:30:25.737545 30277 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.22.53.254@KRBTEST.COM'} at 127.0.0.1:40520:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 07:30:25.738373 30277 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:30:25.744882 30694 tablet_service.cc:1467] Processing CreateTablet for tablet 48e20b422149467d9dfd1fd292bc92fc (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ef1d21398e3943fe9c6e4f6e789d2116]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 07:30:25.744841 30414 tablet_service.cc:1467] Processing CreateTablet for tablet 48e20b422149467d9dfd1fd292bc92fc (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ef1d21398e3943fe9c6e4f6e789d2116]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 07:30:25.744841 30554 tablet_service.cc:1467] Processing CreateTablet for tablet 48e20b422149467d9dfd1fd292bc92fc (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ef1d21398e3943fe9c6e4f6e789d2116]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 07:30:25.745306 30694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48e20b422149467d9dfd1fd292bc92fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:30:25.745301 30414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48e20b422149467d9dfd1fd292bc92fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:30:25.745296 30554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48e20b422149467d9dfd1fd292bc92fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:30:25.747862 30789 tablet_bootstrap.cc:492] T 48e20b422149467d9dfd1fd292bc92fc P 6d6bb899bd634ecd9fb3952a365d22ad: Bootstrap starting.
I20260926 07:30:25.748620 30790 tablet_bootstrap.cc:492] T 48e20b422149467d9dfd1fd292bc92fc P d3cd339e18e549a9b1b07935fe4dd0fc: Bootstrap starting.
I20260926 07:30:25.748874 30789 tablet_bootstrap.cc:654] T 48e20b422149467d9dfd1fd292bc92fc P 6d6bb899bd634ecd9fb3952a365d22ad: Neither blocks nor log segments found. Creating new log.
I20260926 07:30:25.749636 30788 tablet_bootstrap.cc:492] T 48e20b422149467d9dfd1fd292bc92fc P 39b2209b30574a95bf503e074a4f7d29: Bootstrap starting.
I20260926 07:30:25.749648 30790 tablet_bootstrap.cc:654] T 48e20b422149467d9dfd1fd292bc92fc P d3cd339e18e549a9b1b07935fe4dd0fc: Neither blocks nor log segments found. Creating new log.
I20260926 07:30:25.750308 30789 tablet_bootstrap.cc:492] T 48e20b422149467d9dfd1fd292bc92fc P 6d6bb899bd634ecd9fb3952a365d22ad: No bootstrap required, opened a new log
I20260926 07:30:25.750459 30789 ts_tablet_manager.cc:1397] T 48e20b422149467d9dfd1fd292bc92fc P 6d6bb899bd634ecd9fb3952a365d22ad: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 07:30:25.750649 30788 tablet_bootstrap.cc:654] T 48e20b422149467d9dfd1fd292bc92fc P 39b2209b30574a95bf503e074a4f7d29: Neither blocks nor log segments found. Creating new log.
I20260926 07:30:25.750958 30789 raft_consensus.cc:348] T 48e20b422149467d9dfd1fd292bc92fc P 6d6bb899bd634ecd9fb3952a365d22ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39b2209b30574a95bf503e074a4f7d29" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 34137 } } peers { permanent_uuid: "d3cd339e18e549a9b1b07935fe4dd0fc" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 38789 } } peers { permanent_uuid: "6d6bb899bd634ecd9fb3952a365d22ad" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 34909 } }
I20260926 07:30:25.751107 30789 raft_consensus.cc:374] T 48e20b422149467d9dfd1fd292bc92fc P 6d6bb899bd634ecd9fb3952a365d22ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:30:25.751159 30789 raft_consensus.cc:729] T 48e20b422149467d9dfd1fd292bc92fc P 6d6bb899bd634ecd9fb3952a365d22ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d6bb899bd634ecd9fb3952a365d22ad, State: Initialized, Role: FOLLOWER
I20260926 07:30:25.751313 30789 consensus_queue.cc:260] T 48e20b422149467d9dfd1fd292bc92fc P 6d6bb899bd634ecd9fb3952a365d22ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39b2209b30574a95bf503e074a4f7d29" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 34137 } } peers { permanent_uuid: "d3cd339e18e549a9b1b07935fe4dd0fc" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 38789 } } peers { permanent_uuid: "6d6bb899bd634ecd9fb3952a365d22ad" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 34909 } }
I20260926 07:30:25.751662 30789 ts_tablet_manager.cc:1428] T 48e20b422149467d9dfd1fd292bc92fc P 6d6bb899bd634ecd9fb3952a365d22ad: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260926 07:30:25.751761 30798 tablet_replica.cc:439] T 48e20b422149467d9dfd1fd292bc92fc P 6d6bb899bd634ecd9fb3952a365d22ad: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39b2209b30574a95bf503e074a4f7d29" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 34137 } } peers { permanent_uuid: "d3cd339e18e549a9b1b07935fe4dd0fc" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 38789 } } peers { permanent_uuid: "6d6bb899bd634ecd9fb3952a365d22ad" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 34909 } } }
I20260926 07:30:25.751907 30798 tablet_replica.cc:442] T 48e20b422149467d9dfd1fd292bc92fc P 6d6bb899bd634ecd9fb3952a365d22ad: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:30:25.752074 30790 tablet_bootstrap.cc:492] T 48e20b422149467d9dfd1fd292bc92fc P d3cd339e18e549a9b1b07935fe4dd0fc: No bootstrap required, opened a new log
I20260926 07:30:25.752197 30790 ts_tablet_manager.cc:1397] T 48e20b422149467d9dfd1fd292bc92fc P d3cd339e18e549a9b1b07935fe4dd0fc: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260926 07:30:25.752789 30790 raft_consensus.cc:348] T 48e20b422149467d9dfd1fd292bc92fc P d3cd339e18e549a9b1b07935fe4dd0fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39b2209b30574a95bf503e074a4f7d29" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 34137 } } peers { permanent_uuid: "d3cd339e18e549a9b1b07935fe4dd0fc" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 38789 } } peers { permanent_uuid: "6d6bb899bd634ecd9fb3952a365d22ad" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 34909 } }
I20260926 07:30:25.752884 30788 tablet_bootstrap.cc:492] T 48e20b422149467d9dfd1fd292bc92fc P 39b2209b30574a95bf503e074a4f7d29: No bootstrap required, opened a new log
I20260926 07:30:25.752921 30790 raft_consensus.cc:374] T 48e20b422149467d9dfd1fd292bc92fc P d3cd339e18e549a9b1b07935fe4dd0fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:30:25.752956 30790 raft_consensus.cc:729] T 48e20b422149467d9dfd1fd292bc92fc P d3cd339e18e549a9b1b07935fe4dd0fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cd339e18e549a9b1b07935fe4dd0fc, State: Initialized, Role: FOLLOWER
I20260926 07:30:25.753000 30788 ts_tablet_manager.cc:1397] T 48e20b422149467d9dfd1fd292bc92fc P 39b2209b30574a95bf503e074a4f7d29: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 07:30:25.753103 30790 consensus_queue.cc:260] T 48e20b422149467d9dfd1fd292bc92fc P d3cd339e18e549a9b1b07935fe4dd0fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39b2209b30574a95bf503e074a4f7d29" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 34137 } } peers { permanent_uuid: "d3cd339e18e549a9b1b07935fe4dd0fc" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 38789 } } peers { permanent_uuid: "6d6bb899bd634ecd9fb3952a365d22ad" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 34909 } }
I20260926 07:30:25.753414 30790 ts_tablet_manager.cc:1428] T 48e20b422149467d9dfd1fd292bc92fc P d3cd339e18e549a9b1b07935fe4dd0fc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 07:30:25.753400 30796 tablet_replica.cc:439] T 48e20b422149467d9dfd1fd292bc92fc P d3cd339e18e549a9b1b07935fe4dd0fc: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39b2209b30574a95bf503e074a4f7d29" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 34137 } } peers { permanent_uuid: "d3cd339e18e549a9b1b07935fe4dd0fc" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 38789 } } peers { permanent_uuid: "6d6bb899bd634ecd9fb3952a365d22ad" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 34909 } } }
I20260926 07:30:25.753568 30796 tablet_replica.cc:442] T 48e20b422149467d9dfd1fd292bc92fc P d3cd339e18e549a9b1b07935fe4dd0fc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:30:25.753516 30788 raft_consensus.cc:348] T 48e20b422149467d9dfd1fd292bc92fc P 39b2209b30574a95bf503e074a4f7d29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39b2209b30574a95bf503e074a4f7d29" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 34137 } } peers { permanent_uuid: "d3cd339e18e549a9b1b07935fe4dd0fc" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 38789 } } peers { permanent_uuid: "6d6bb899bd634ecd9fb3952a365d22ad" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 34909 } }
I20260926 07:30:25.753633 30788 raft_consensus.cc:374] T 48e20b422149467d9dfd1fd292bc92fc P 39b2209b30574a95bf503e074a4f7d29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:30:25.753703 30788 raft_consensus.cc:729] T 48e20b422149467d9dfd1fd292bc92fc P 39b2209b30574a95bf503e074a4f7d29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39b2209b30574a95bf503e074a4f7d29, State: Initialized, Role: FOLLOWER
I20260926 07:30:25.753840 30788 consensus_queue.cc:260] T 48e20b422149467d9dfd1fd292bc92fc P 39b2209b30574a95bf503e074a4f7d29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39b2209b30574a95bf503e074a4f7d29" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 34137 } } peers { permanent_uuid: "d3cd339e18e549a9b1b07935fe4dd0fc" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 38789 } } peers { permanent_uuid: "6d6bb899bd634ecd9fb3952a365d22ad" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 34909 } }
I20260926 07:30:25.754158 30788 ts_tablet_manager.cc:1428] T 48e20b422149467d9dfd1fd292bc92fc P 39b2209b30574a95bf503e074a4f7d29: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 07:30:25.754142 30795 tablet_replica.cc:439] T 48e20b422149467d9dfd1fd292bc92fc P 39b2209b30574a95bf503e074a4f7d29: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39b2209b30574a95bf503e074a4f7d29" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 34137 } } peers { permanent_uuid: "d3cd339e18e549a9b1b07935fe4dd0fc" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 38789 } } peers { permanent_uuid: "6d6bb899bd634ecd9fb3952a365d22ad" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 34909 } } }
I20260926 07:30:25.754312 30795 tablet_replica.cc:442] T 48e20b422149467d9dfd1fd292bc92fc P 39b2209b30574a95bf503e074a4f7d29: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:30:25.828828 30796 raft_consensus.cc:482] T 48e20b422149467d9dfd1fd292bc92fc P d3cd339e18e549a9b1b07935fe4dd0fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:30:25.828994 30796 raft_consensus.cc:504] T 48e20b422149467d9dfd1fd292bc92fc P d3cd339e18e549a9b1b07935fe4dd0fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39b2209b30574a95bf503e074a4f7d29" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 34137 } } peers { permanent_uuid: "d3cd339e18e549a9b1b07935fe4dd0fc" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 38789 } } peers { permanent_uuid: "6d6bb899bd634ecd9fb3952a365d22ad" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 34909 } }
I20260926 07:30:25.829586 30796 leader_election.cc:288] T 48e20b422149467d9dfd1fd292bc92fc P d3cd339e18e549a9b1b07935fe4dd0fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39b2209b30574a95bf503e074a4f7d29 (127.22.53.193:34137), 6d6bb899bd634ecd9fb3952a365d22ad (127.22.53.195:34909)
I20260926 07:30:25.830107 30434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48e20b422149467d9dfd1fd292bc92fc" candidate_uuid: "d3cd339e18e549a9b1b07935fe4dd0fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39b2209b30574a95bf503e074a4f7d29" is_pre_election: true
I20260926 07:30:25.830256 30434 raft_consensus.cc:2454] T 48e20b422149467d9dfd1fd292bc92fc P 39b2209b30574a95bf503e074a4f7d29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3cd339e18e549a9b1b07935fe4dd0fc in term 0.
I20260926 07:30:25.830269 30714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48e20b422149467d9dfd1fd292bc92fc" candidate_uuid: "d3cd339e18e549a9b1b07935fe4dd0fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d6bb899bd634ecd9fb3952a365d22ad" is_pre_election: true
I20260926 07:30:25.830500 30714 raft_consensus.cc:2454] T 48e20b422149467d9dfd1fd292bc92fc P 6d6bb899bd634ecd9fb3952a365d22ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3cd339e18e549a9b1b07935fe4dd0fc in term 0.
I20260926 07:30:25.830614 30507 leader_election.cc:302] T 48e20b422149467d9dfd1fd292bc92fc P d3cd339e18e549a9b1b07935fe4dd0fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39b2209b30574a95bf503e074a4f7d29, d3cd339e18e549a9b1b07935fe4dd0fc; no voters:
I20260926 07:30:25.830878 30796 raft_consensus.cc:2781] T 48e20b422149467d9dfd1fd292bc92fc P d3cd339e18e549a9b1b07935fe4dd0fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:30:25.830960 30796 raft_consensus.cc:482] T 48e20b422149467d9dfd1fd292bc92fc P d3cd339e18e549a9b1b07935fe4dd0fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:30:25.831015 30796 raft_consensus.cc:3037] T 48e20b422149467d9dfd1fd292bc92fc P d3cd339e18e549a9b1b07935fe4dd0fc [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:30:25.832301 30796 raft_consensus.cc:504] T 48e20b422149467d9dfd1fd292bc92fc P d3cd339e18e549a9b1b07935fe4dd0fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39b2209b30574a95bf503e074a4f7d29" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 34137 } } peers { permanent_uuid: "d3cd339e18e549a9b1b07935fe4dd0fc" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 38789 } } peers { permanent_uuid: "6d6bb899bd634ecd9fb3952a365d22ad" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 34909 } }
I20260926 07:30:25.832756 30796 leader_election.cc:288] T 48e20b422149467d9dfd1fd292bc92fc P d3cd339e18e549a9b1b07935fe4dd0fc [CANDIDATE]: Term 1 election: Requested vote from peers 39b2209b30574a95bf503e074a4f7d29 (127.22.53.193:34137), 6d6bb899bd634ecd9fb3952a365d22ad (127.22.53.195:34909)
I20260926 07:30:25.833241 30714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48e20b422149467d9dfd1fd292bc92fc" candidate_uuid: "d3cd339e18e549a9b1b07935fe4dd0fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d6bb899bd634ecd9fb3952a365d22ad"
I20260926 07:30:25.833264 30434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48e20b422149467d9dfd1fd292bc92fc" candidate_uuid: "d3cd339e18e549a9b1b07935fe4dd0fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39b2209b30574a95bf503e074a4f7d29"
I20260926 07:30:25.833372 30714 raft_consensus.cc:3037] T 48e20b422149467d9dfd1fd292bc92fc P 6d6bb899bd634ecd9fb3952a365d22ad [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:30:25.833424 30434 raft_consensus.cc:3037] T 48e20b422149467d9dfd1fd292bc92fc P 39b2209b30574a95bf503e074a4f7d29 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:30:25.834465 30714 raft_consensus.cc:2454] T 48e20b422149467d9dfd1fd292bc92fc P 6d6bb899bd634ecd9fb3952a365d22ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3cd339e18e549a9b1b07935fe4dd0fc in term 1.
I20260926 07:30:25.834461 30434 raft_consensus.cc:2454] T 48e20b422149467d9dfd1fd292bc92fc P 39b2209b30574a95bf503e074a4f7d29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3cd339e18e549a9b1b07935fe4dd0fc in term 1.
I20260926 07:30:25.834854 30507 leader_election.cc:302] T 48e20b422149467d9dfd1fd292bc92fc P d3cd339e18e549a9b1b07935fe4dd0fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39b2209b30574a95bf503e074a4f7d29, d3cd339e18e549a9b1b07935fe4dd0fc; no voters:
I20260926 07:30:25.835088 30796 raft_consensus.cc:2781] T 48e20b422149467d9dfd1fd292bc92fc P d3cd339e18e549a9b1b07935fe4dd0fc [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:30:25.835182 30796 raft_consensus.cc:686] T 48e20b422149467d9dfd1fd292bc92fc P d3cd339e18e549a9b1b07935fe4dd0fc [term 1 LEADER]: Becoming Leader. State: Replica: d3cd339e18e549a9b1b07935fe4dd0fc, State: Running, Role: LEADER
I20260926 07:30:25.835371 30796 consensus_queue.cc:237] T 48e20b422149467d9dfd1fd292bc92fc P d3cd339e18e549a9b1b07935fe4dd0fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39b2209b30574a95bf503e074a4f7d29" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 34137 } } peers { permanent_uuid: "d3cd339e18e549a9b1b07935fe4dd0fc" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 38789 } } peers { permanent_uuid: "6d6bb899bd634ecd9fb3952a365d22ad" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 34909 } }
I20260926 07:30:25.836203 30819 tablet_replica.cc:439] T 48e20b422149467d9dfd1fd292bc92fc P d3cd339e18e549a9b1b07935fe4dd0fc: TxnStatusTablet state changed. Reason: New leader d3cd339e18e549a9b1b07935fe4dd0fc. Latest consensus state: current_term: 1 leader_uuid: "d3cd339e18e549a9b1b07935fe4dd0fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39b2209b30574a95bf503e074a4f7d29" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 34137 } } peers { permanent_uuid: "d3cd339e18e549a9b1b07935fe4dd0fc" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 38789 } } peers { permanent_uuid: "6d6bb899bd634ecd9fb3952a365d22ad" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 34909 } } }
I20260926 07:30:25.836341 30819 tablet_replica.cc:442] T 48e20b422149467d9dfd1fd292bc92fc P d3cd339e18e549a9b1b07935fe4dd0fc: This TxnStatusTablet replica's current role is: LEADER
I20260926 07:30:25.836736 30831 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 07:30:25.836928 30831 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 07:30:25.837529 30276 catalog_manager.cc:5495] T 48e20b422149467d9dfd1fd292bc92fc P d3cd339e18e549a9b1b07935fe4dd0fc reported cstate change: term changed from 0 to 1, leader changed from <none> to d3cd339e18e549a9b1b07935fe4dd0fc (127.22.53.194). New cstate: current_term: 1 leader_uuid: "d3cd339e18e549a9b1b07935fe4dd0fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39b2209b30574a95bf503e074a4f7d29" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 34137 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3cd339e18e549a9b1b07935fe4dd0fc" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 38789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6d6bb899bd634ecd9fb3952a365d22ad" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 34909 } health_report { overall_health: UNKNOWN } } }
I20260926 07:30:25.862515 30826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:25.858121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.194:38789 (local address 127.0.0.1:36132)
0926 07:30:25.858328 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:25.858345 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:25.858484 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:25.858837 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:25.858840 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:25.858846 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 07:30:25.858967 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:25.858973 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:25.859704 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:25.859711 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:25.860421 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:25.860435 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:25.861151 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:25.861153 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:25.861225 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:25.861286 (+ 61us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 07:30:25.862189 (+ 903us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 07:30:25.862206 (+ 17us) client_negotiation.cc:769] Sending connection context
0926 07:30:25.862288 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 07:30:25.862380 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75}
I20260926 07:30:25.862541 30787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:25.858236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36132 (local address 127.22.53.194:38789)
0926 07:30:25.858367 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:25.858374 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:25.858395 (+ 21us) server_negotiation.cc:407] Connection header received
0926 07:30:25.858657 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:25.858660 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:25.858716 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:25.858824 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:30:25.859081 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:25.859587 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:25.860603 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:25.861016 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:25.861145 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:25.861447 (+ 302us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:30:25.862070 (+ 623us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:30:25.862200 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:25.862310 (+ 110us) server_negotiation.cc:299] Negotiation successful
0926 07:30:25.862380 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 07:30:25.866058 30434 raft_consensus.cc:1261] T 48e20b422149467d9dfd1fd292bc92fc P 39b2209b30574a95bf503e074a4f7d29 [term 1 FOLLOWER]: Refusing update from remote peer d3cd339e18e549a9b1b07935fe4dd0fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:30:25.866345 30714 raft_consensus.cc:1261] T 48e20b422149467d9dfd1fd292bc92fc P 6d6bb899bd634ecd9fb3952a365d22ad [term 1 FOLLOWER]: Refusing update from remote peer d3cd339e18e549a9b1b07935fe4dd0fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:30:25.866708 30819 consensus_queue.cc:1035] T 48e20b422149467d9dfd1fd292bc92fc P d3cd339e18e549a9b1b07935fe4dd0fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "39b2209b30574a95bf503e074a4f7d29" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 34137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:30:25.867322 30819 consensus_queue.cc:1035] T 48e20b422149467d9dfd1fd292bc92fc P d3cd339e18e549a9b1b07935fe4dd0fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d6bb899bd634ecd9fb3952a365d22ad" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 34909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:30:25.868774 30795 tablet_replica.cc:439] T 48e20b422149467d9dfd1fd292bc92fc P 39b2209b30574a95bf503e074a4f7d29: TxnStatusTablet state changed. Reason: New leader d3cd339e18e549a9b1b07935fe4dd0fc. Latest consensus state: current_term: 1 leader_uuid: "d3cd339e18e549a9b1b07935fe4dd0fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39b2209b30574a95bf503e074a4f7d29" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 34137 } } peers { permanent_uuid: "d3cd339e18e549a9b1b07935fe4dd0fc" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 38789 } } peers { permanent_uuid: "6d6bb899bd634ecd9fb3952a365d22ad" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 34909 } } }
I20260926 07:30:25.868942 30795 tablet_replica.cc:442] T 48e20b422149467d9dfd1fd292bc92fc P 39b2209b30574a95bf503e074a4f7d29: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:30:25.869256 30798 tablet_replica.cc:439] T 48e20b422149467d9dfd1fd292bc92fc P 6d6bb899bd634ecd9fb3952a365d22ad: TxnStatusTablet state changed. Reason: New leader d3cd339e18e549a9b1b07935fe4dd0fc. Latest consensus state: current_term: 1 leader_uuid: "d3cd339e18e549a9b1b07935fe4dd0fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39b2209b30574a95bf503e074a4f7d29" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 34137 } } peers { permanent_uuid: "d3cd339e18e549a9b1b07935fe4dd0fc" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 38789 } } peers { permanent_uuid: "6d6bb899bd634ecd9fb3952a365d22ad" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 34909 } } }
I20260926 07:30:25.869413 30798 tablet_replica.cc:442] T 48e20b422149467d9dfd1fd292bc92fc P 6d6bb899bd634ecd9fb3952a365d22ad: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:30:25.872941 30803 mvcc.cc:205] Tried to move back new op lower bound from 7333510454741725184 to 7333510454623252480. Current Snapshot: MvccSnapshot[applied={T|T < 7333510454741725184}]
I20260926 07:30:25.873060 30798 tablet_replica.cc:439] T 48e20b422149467d9dfd1fd292bc92fc P 6d6bb899bd634ecd9fb3952a365d22ad: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d3cd339e18e549a9b1b07935fe4dd0fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39b2209b30574a95bf503e074a4f7d29" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 34137 } } peers { permanent_uuid: "d3cd339e18e549a9b1b07935fe4dd0fc" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 38789 } } peers { permanent_uuid: "6d6bb899bd634ecd9fb3952a365d22ad" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 34909 } } }
I20260926 07:30:25.873203 30798 tablet_replica.cc:442] T 48e20b422149467d9dfd1fd292bc92fc P 6d6bb899bd634ecd9fb3952a365d22ad: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:30:25.873327 30795 tablet_replica.cc:439] T 48e20b422149467d9dfd1fd292bc92fc P 39b2209b30574a95bf503e074a4f7d29: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d3cd339e18e549a9b1b07935fe4dd0fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39b2209b30574a95bf503e074a4f7d29" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 34137 } } peers { permanent_uuid: "d3cd339e18e549a9b1b07935fe4dd0fc" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 38789 } } peers { permanent_uuid: "6d6bb899bd634ecd9fb3952a365d22ad" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 34909 } } }
I20260926 07:30:25.873450 30795 tablet_replica.cc:442] T 48e20b422149467d9dfd1fd292bc92fc P 39b2209b30574a95bf503e074a4f7d29: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:30:25.872524 30796 tablet_replica.cc:439] T 48e20b422149467d9dfd1fd292bc92fc P d3cd339e18e549a9b1b07935fe4dd0fc: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d3cd339e18e549a9b1b07935fe4dd0fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39b2209b30574a95bf503e074a4f7d29" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 34137 } } peers { permanent_uuid: "d3cd339e18e549a9b1b07935fe4dd0fc" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 38789 } } peers { permanent_uuid: "6d6bb899bd634ecd9fb3952a365d22ad" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 34909 } } }
I20260926 07:30:25.875556 30796 tablet_replica.cc:442] T 48e20b422149467d9dfd1fd292bc92fc P d3cd339e18e549a9b1b07935fe4dd0fc: This TxnStatusTablet replica's current role is: LEADER
I20260926 07:30:25.875737 30796 tablet_replica.cc:439] T 48e20b422149467d9dfd1fd292bc92fc P d3cd339e18e549a9b1b07935fe4dd0fc: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d3cd339e18e549a9b1b07935fe4dd0fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39b2209b30574a95bf503e074a4f7d29" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 34137 } } peers { permanent_uuid: "d3cd339e18e549a9b1b07935fe4dd0fc" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 38789 } } peers { permanent_uuid: "6d6bb899bd634ecd9fb3952a365d22ad" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 34909 } } }
I20260926 07:30:25.875833 30796 tablet_replica.cc:442] T 48e20b422149467d9dfd1fd292bc92fc P d3cd339e18e549a9b1b07935fe4dd0fc: This TxnStatusTablet replica's current role is: LEADER
I20260926 07:30:25.883080 30787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:25.879313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36136 (local address 127.22.53.194:38789)
0926 07:30:25.879440 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:25.879445 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:25.879460 (+ 15us) server_negotiation.cc:407] Connection header received
0926 07:30:25.879547 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:25.879550 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:25.879601 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:25.879685 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:30:25.879913 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:25.880767 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:25.881759 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:25.882139 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:25.882261 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:25.882491 (+ 230us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:30:25.882589 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:30:25.882669 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:25.882858 (+ 189us) server_negotiation.cc:299] Negotiation successful
0926 07:30:25.882924 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 07:30:25.896318 30787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:25.891870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.194:49075 (local address 127.22.53.194:38789)
0926 07:30:25.892009 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:25.892013 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:25.892759 (+ 746us) server_negotiation.cc:407] Connection header received
0926 07:30:25.893138 (+ 379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:25.893146 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:25.893193 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:25.893280 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:30:25.893608 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:25.894114 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:25.895136 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:25.895347 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:25.895459 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:25.895799 (+ 340us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:30:25.895876 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:30:25.895971 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:25.896059 (+ 88us) server_negotiation.cc:299] Negotiation successful
0926 07:30:25.896144 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":63,"server-negotiator.queue_time_us":51}
I20260926 07:30:25.896323 30839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:25.891768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.194:38789 (local address 127.22.53.194:49075)
0926 07:30:25.892664 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:25.892679 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:25.892831 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:25.893286 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:25.893289 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:25.893296 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 07:30:25.893495 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:25.893502 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:25.894224 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:25.894230 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:25.895022 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:25.895032 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:25.895472 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:25.895475 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:25.895583 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:25.895640 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 07:30:25.895974 (+ 334us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 07:30:25.895982 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 07:30:25.896044 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 07:30:25.896157 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":740,"mutex_wait_us":41,"thread_start_us":624,"threads_started":1}
I20260926 07:30:25.904088 30434 raft_consensus.cc:1261] T 0545bdb9f17e41ec95c8c2e9df8561be P 39b2209b30574a95bf503e074a4f7d29 [term 1 FOLLOWER]: Refusing update from remote peer d3cd339e18e549a9b1b07935fe4dd0fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:30:25.904122 30714 raft_consensus.cc:1261] T 0545bdb9f17e41ec95c8c2e9df8561be P 6d6bb899bd634ecd9fb3952a365d22ad [term 1 FOLLOWER]: Refusing update from remote peer d3cd339e18e549a9b1b07935fe4dd0fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:30:25.904592 30819 consensus_queue.cc:1035] T 0545bdb9f17e41ec95c8c2e9df8561be P d3cd339e18e549a9b1b07935fe4dd0fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "39b2209b30574a95bf503e074a4f7d29" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 34137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:30:25.905136 30819 consensus_queue.cc:1035] T 0545bdb9f17e41ec95c8c2e9df8561be P d3cd339e18e549a9b1b07935fe4dd0fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d6bb899bd634ecd9fb3952a365d22ad" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 34909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:30:25.959932 30276 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:30:25.960137 30276 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:30:25.963047 30276 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 945929f517c847deaee9a27a48db5aa9: Sending DeleteTablet for 3 replicas of tablet 0545bdb9f17e41ec95c8c2e9df8561be
I20260926 07:30:25.964015 30553 tablet_service.cc:1514] Processing DeleteTablet for tablet 0545bdb9f17e41ec95c8c2e9df8561be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:30:25 UTC) from {username='oryx', principal='oryx/127.22.53.254@KRBTEST.COM'} at 127.0.0.1:36126
I20260926 07:30:25.964130 30414 tablet_service.cc:1514] Processing DeleteTablet for tablet 0545bdb9f17e41ec95c8c2e9df8561be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:30:25 UTC) from {username='oryx', principal='oryx/127.22.53.254@KRBTEST.COM'} at 127.0.0.1:42458
I20260926 07:30:25.964322 30694 tablet_service.cc:1514] Processing DeleteTablet for tablet 0545bdb9f17e41ec95c8c2e9df8561be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:30:25 UTC) from {username='oryx', principal='oryx/127.22.53.254@KRBTEST.COM'} at 127.0.0.1:59978
I20260926 07:30:25.964411 30844 tablet_replica.cc:330] T 0545bdb9f17e41ec95c8c2e9df8561be P d3cd339e18e549a9b1b07935fe4dd0fc: stopping tablet replica
I20260926 07:30:25.964564 30812 tablet_replica.cc:330] T 0545bdb9f17e41ec95c8c2e9df8561be P 6d6bb899bd634ecd9fb3952a365d22ad: stopping tablet replica
I20260926 07:30:25.964574 30844 raft_consensus.cc:2229] T 0545bdb9f17e41ec95c8c2e9df8561be P d3cd339e18e549a9b1b07935fe4dd0fc [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:30:25.964712 30812 raft_consensus.cc:2229] T 0545bdb9f17e41ec95c8c2e9df8561be P 6d6bb899bd634ecd9fb3952a365d22ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:30:25.964856 30812 raft_consensus.cc:2258] T 0545bdb9f17e41ec95c8c2e9df8561be P 6d6bb899bd634ecd9fb3952a365d22ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:30:25.964936 30844 raft_consensus.cc:2258] T 0545bdb9f17e41ec95c8c2e9df8561be P d3cd339e18e549a9b1b07935fe4dd0fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:30:25.965032 30845 tablet_replica.cc:330] T 0545bdb9f17e41ec95c8c2e9df8561be P 39b2209b30574a95bf503e074a4f7d29: stopping tablet replica
I20260926 07:30:25.965194 30845 raft_consensus.cc:2229] T 0545bdb9f17e41ec95c8c2e9df8561be P 39b2209b30574a95bf503e074a4f7d29 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:30:25.965365 30845 raft_consensus.cc:2258] T 0545bdb9f17e41ec95c8c2e9df8561be P 39b2209b30574a95bf503e074a4f7d29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:30:25.965816 30844 ts_tablet_manager.cc:1905] T 0545bdb9f17e41ec95c8c2e9df8561be P d3cd339e18e549a9b1b07935fe4dd0fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:30:25.966184 30845 ts_tablet_manager.cc:1905] T 0545bdb9f17e41ec95c8c2e9df8561be P 39b2209b30574a95bf503e074a4f7d29: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:30:25.969425 30845 ts_tablet_manager.cc:1918] T 0545bdb9f17e41ec95c8c2e9df8561be P 39b2209b30574a95bf503e074a4f7d29: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 07:30:25.969425 30844 ts_tablet_manager.cc:1918] T 0545bdb9f17e41ec95c8c2e9df8561be P d3cd339e18e549a9b1b07935fe4dd0fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 07:30:25.969579 30845 log.cc:1196] T 0545bdb9f17e41ec95c8c2e9df8561be P 39b2209b30574a95bf503e074a4f7d29: Deleting WAL directory at /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407783529717-22743-0/minicluster-data/ts-0/wal/wals/0545bdb9f17e41ec95c8c2e9df8561be
I20260926 07:30:25.969583 30812 ts_tablet_manager.cc:1905] T 0545bdb9f17e41ec95c8c2e9df8561be P 6d6bb899bd634ecd9fb3952a365d22ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:30:25.969524 30844 log.cc:1196] T 0545bdb9f17e41ec95c8c2e9df8561be P d3cd339e18e549a9b1b07935fe4dd0fc: Deleting WAL directory at /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407783529717-22743-0/minicluster-data/ts-1/wal/wals/0545bdb9f17e41ec95c8c2e9df8561be
I20260926 07:30:25.970000 30845 ts_tablet_manager.cc:1939] T 0545bdb9f17e41ec95c8c2e9df8561be P 39b2209b30574a95bf503e074a4f7d29: Deleting consensus metadata
I20260926 07:30:25.970041 30844 ts_tablet_manager.cc:1939] T 0545bdb9f17e41ec95c8c2e9df8561be P d3cd339e18e549a9b1b07935fe4dd0fc: Deleting consensus metadata
I20260926 07:30:25.970974 30263 catalog_manager.cc:4841] TS 39b2209b30574a95bf503e074a4f7d29 (127.22.53.193:34137): tablet 0545bdb9f17e41ec95c8c2e9df8561be (table test-table [id=864b71dfdf7c40d9bc69f2ddc318a8a6]) successfully deleted
I20260926 07:30:25.970974 30262 catalog_manager.cc:4841] TS d3cd339e18e549a9b1b07935fe4dd0fc (127.22.53.194:38789): tablet 0545bdb9f17e41ec95c8c2e9df8561be (table test-table [id=864b71dfdf7c40d9bc69f2ddc318a8a6]) successfully deleted
I20260926 07:30:25.972836 30812 ts_tablet_manager.cc:1918] T 0545bdb9f17e41ec95c8c2e9df8561be P 6d6bb899bd634ecd9fb3952a365d22ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 07:30:25.972942 30812 log.cc:1196] T 0545bdb9f17e41ec95c8c2e9df8561be P 6d6bb899bd634ecd9fb3952a365d22ad: Deleting WAL directory at /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407783529717-22743-0/minicluster-data/ts-2/wal/wals/0545bdb9f17e41ec95c8c2e9df8561be
I20260926 07:30:25.973201 30812 ts_tablet_manager.cc:1939] T 0545bdb9f17e41ec95c8c2e9df8561be P 6d6bb899bd634ecd9fb3952a365d22ad: Deleting consensus metadata
I20260926 07:30:25.974167 30261 catalog_manager.cc:4841] TS 6d6bb899bd634ecd9fb3952a365d22ad (127.22.53.195:34909): tablet 0545bdb9f17e41ec95c8c2e9df8561be (table test-table [id=864b71dfdf7c40d9bc69f2ddc318a8a6]) successfully deleted
Sep 26 07:30:25 dist-test-slave-wdjc krb5kdc[30232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407825, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 07:30:25 dist-test-slave-wdjc krb5kdc[30232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.22.53.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:30:25 dist-test-slave-wdjc krb5kdc[30232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.22.53.254@KRBTEST.COM, Server not found in Kerberos database
W20260926 07:30:26.001797 30326 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:30:25.993892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40528 (local address 127.22.53.254:39267)
0926 07:30:25.994103 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:25.994110 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:25.994366 (+ 256us) server_negotiation.cc:407] Connection header received
0926 07:30:25.994592 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:25.994597 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:25.994656 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:25.994805 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:25.995785 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:25.996391 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:25.997499 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:25.997774 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:25.997921 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:26.001661 (+ 3740us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40528: BlockingRecv error: recv got EOF from 127.0.0.1:40528 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 26 07:30:26 dist-test-slave-wdjc krb5kdc[30232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407825, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM
I20260926 07:30:26.013876 30326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:26.003806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40530 (local address 127.22.53.254:39267)
0926 07:30:26.003928 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:26.003933 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:26.004010 (+ 77us) server_negotiation.cc:407] Connection header received
0926 07:30:26.004155 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:26.004159 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:26.004220 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:26.004329 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:26.005105 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:26.005934 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:26.006825 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:26.007159 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:26.007301 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:26.010107 (+ 2806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:26.010112 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:26.010115 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:26.010154 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:26.011861 (+ 1707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:26.012410 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:26.012413 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:26.012415 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:26.012520 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:26.012900 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:26.012904 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:26.012905 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:26.013164 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:26.013301 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:26.013573 (+ 272us) server_negotiation.cc:299] Negotiation successful
0926 07:30:26.013701 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 07:30:26.017778 30276 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40530:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:30:26.018299 30276 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:30:26.024938 30553 tablet_service.cc:1467] Processing CreateTablet for tablet 10becadd92c14019928d65c704724798 (DEFAULT_TABLE table=test-table [id=8a89c828871546d68b12432352b1ed8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:30:26.025302 30553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10becadd92c14019928d65c704724798. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:30:26.026517 30414 tablet_service.cc:1467] Processing CreateTablet for tablet 10becadd92c14019928d65c704724798 (DEFAULT_TABLE table=test-table [id=8a89c828871546d68b12432352b1ed8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:30:26.026901 30414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10becadd92c14019928d65c704724798. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:30:26.029832 30790 tablet_bootstrap.cc:492] T 10becadd92c14019928d65c704724798 P d3cd339e18e549a9b1b07935fe4dd0fc: Bootstrap starting.
I20260926 07:30:26.030747 30790 tablet_bootstrap.cc:654] T 10becadd92c14019928d65c704724798 P d3cd339e18e549a9b1b07935fe4dd0fc: Neither blocks nor log segments found. Creating new log.
I20260926 07:30:26.031302 30788 tablet_bootstrap.cc:492] T 10becadd92c14019928d65c704724798 P 39b2209b30574a95bf503e074a4f7d29: Bootstrap starting.
I20260926 07:30:26.032248 30788 tablet_bootstrap.cc:654] T 10becadd92c14019928d65c704724798 P 39b2209b30574a95bf503e074a4f7d29: Neither blocks nor log segments found. Creating new log.
I20260926 07:30:26.033352 30694 tablet_service.cc:1467] Processing CreateTablet for tablet 10becadd92c14019928d65c704724798 (DEFAULT_TABLE table=test-table [id=8a89c828871546d68b12432352b1ed8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:30:26.033614 30790 tablet_bootstrap.cc:492] T 10becadd92c14019928d65c704724798 P d3cd339e18e549a9b1b07935fe4dd0fc: No bootstrap required, opened a new log
I20260926 07:30:26.033710 30790 ts_tablet_manager.cc:1397] T 10becadd92c14019928d65c704724798 P d3cd339e18e549a9b1b07935fe4dd0fc: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260926 07:30:26.033720 30694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10becadd92c14019928d65c704724798. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:30:26.034159 30790 raft_consensus.cc:348] T 10becadd92c14019928d65c704724798 P d3cd339e18e549a9b1b07935fe4dd0fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39b2209b30574a95bf503e074a4f7d29" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 34137 } } peers { permanent_uuid: "d3cd339e18e549a9b1b07935fe4dd0fc" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 38789 } } peers { permanent_uuid: "6d6bb899bd634ecd9fb3952a365d22ad" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 34909 } }
I20260926 07:30:26.034282 30790 raft_consensus.cc:374] T 10becadd92c14019928d65c704724798 P d3cd339e18e549a9b1b07935fe4dd0fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:30:26.034319 30790 raft_consensus.cc:729] T 10becadd92c14019928d65c704724798 P d3cd339e18e549a9b1b07935fe4dd0fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cd339e18e549a9b1b07935fe4dd0fc, State: Initialized, Role: FOLLOWER
I20260926 07:30:26.034451 30790 consensus_queue.cc:260] T 10becadd92c14019928d65c704724798 P d3cd339e18e549a9b1b07935fe4dd0fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39b2209b30574a95bf503e074a4f7d29" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 34137 } } peers { permanent_uuid: "d3cd339e18e549a9b1b07935fe4dd0fc" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 38789 } } peers { permanent_uuid: "6d6bb899bd634ecd9fb3952a365d22ad" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 34909 } }
I20260926 07:30:26.034684 30790 ts_tablet_manager.cc:1428] T 10becadd92c14019928d65c704724798 P d3cd339e18e549a9b1b07935fe4dd0fc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 07:30:26.036680 30788 tablet_bootstrap.cc:492] T 10becadd92c14019928d65c704724798 P 39b2209b30574a95bf503e074a4f7d29: No bootstrap required, opened a new log
I20260926 07:30:26.036777 30788 ts_tablet_manager.cc:1397] T 10becadd92c14019928d65c704724798 P 39b2209b30574a95bf503e074a4f7d29: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20260926 07:30:26.037125 30789 tablet_bootstrap.cc:492] T 10becadd92c14019928d65c704724798 P 6d6bb899bd634ecd9fb3952a365d22ad: Bootstrap starting.
I20260926 07:30:26.037257 30788 raft_consensus.cc:348] T 10becadd92c14019928d65c704724798 P 39b2209b30574a95bf503e074a4f7d29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39b2209b30574a95bf503e074a4f7d29" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 34137 } } peers { permanent_uuid: "d3cd339e18e549a9b1b07935fe4dd0fc" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 38789 } } peers { permanent_uuid: "6d6bb899bd634ecd9fb3952a365d22ad" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 34909 } }
I20260926 07:30:26.037370 30788 raft_consensus.cc:374] T 10becadd92c14019928d65c704724798 P 39b2209b30574a95bf503e074a4f7d29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:30:26.037400 30788 raft_consensus.cc:729] T 10becadd92c14019928d65c704724798 P 39b2209b30574a95bf503e074a4f7d29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39b2209b30574a95bf503e074a4f7d29, State: Initialized, Role: FOLLOWER
I20260926 07:30:26.037513 30788 consensus_queue.cc:260] T 10becadd92c14019928d65c704724798 P 39b2209b30574a95bf503e074a4f7d29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39b2209b30574a95bf503e074a4f7d29" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 34137 } } peers { permanent_uuid: "d3cd339e18e549a9b1b07935fe4dd0fc" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 38789 } } peers { permanent_uuid: "6d6bb899bd634ecd9fb3952a365d22ad" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 34909 } }
I20260926 07:30:26.037771 30788 ts_tablet_manager.cc:1428] T 10becadd92c14019928d65c704724798 P 39b2209b30574a95bf503e074a4f7d29: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260926 07:30:26.038208 30789 tablet_bootstrap.cc:654] T 10becadd92c14019928d65c704724798 P 6d6bb899bd634ecd9fb3952a365d22ad: Neither blocks nor log segments found. Creating new log.
I20260926 07:30:26.040611 30789 tablet_bootstrap.cc:492] T 10becadd92c14019928d65c704724798 P 6d6bb899bd634ecd9fb3952a365d22ad: No bootstrap required, opened a new log
I20260926 07:30:26.040715 30789 ts_tablet_manager.cc:1397] T 10becadd92c14019928d65c704724798 P 6d6bb899bd634ecd9fb3952a365d22ad: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260926 07:30:26.041131 30789 raft_consensus.cc:348] T 10becadd92c14019928d65c704724798 P 6d6bb899bd634ecd9fb3952a365d22ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39b2209b30574a95bf503e074a4f7d29" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 34137 } } peers { permanent_uuid: "d3cd339e18e549a9b1b07935fe4dd0fc" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 38789 } } peers { permanent_uuid: "6d6bb899bd634ecd9fb3952a365d22ad" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 34909 } }
I20260926 07:30:26.041261 30789 raft_consensus.cc:374] T 10becadd92c14019928d65c704724798 P 6d6bb899bd634ecd9fb3952a365d22ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:30:26.041301 30789 raft_consensus.cc:729] T 10becadd92c14019928d65c704724798 P 6d6bb899bd634ecd9fb3952a365d22ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d6bb899bd634ecd9fb3952a365d22ad, State: Initialized, Role: FOLLOWER
I20260926 07:30:26.041442 30789 consensus_queue.cc:260] T 10becadd92c14019928d65c704724798 P 6d6bb899bd634ecd9fb3952a365d22ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39b2209b30574a95bf503e074a4f7d29" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 34137 } } peers { permanent_uuid: "d3cd339e18e549a9b1b07935fe4dd0fc" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 38789 } } peers { permanent_uuid: "6d6bb899bd634ecd9fb3952a365d22ad" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 34909 } }
I20260926 07:30:26.041743 30789 ts_tablet_manager.cc:1428] T 10becadd92c14019928d65c704724798 P 6d6bb899bd634ecd9fb3952a365d22ad: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260926 07:30:26.142001 30795 raft_consensus.cc:482] T 10becadd92c14019928d65c704724798 P 39b2209b30574a95bf503e074a4f7d29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:30:26.142163 30795 raft_consensus.cc:504] T 10becadd92c14019928d65c704724798 P 39b2209b30574a95bf503e074a4f7d29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39b2209b30574a95bf503e074a4f7d29" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 34137 } } peers { permanent_uuid: "d3cd339e18e549a9b1b07935fe4dd0fc" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 38789 } } peers { permanent_uuid: "6d6bb899bd634ecd9fb3952a365d22ad" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 34909 } }
I20260926 07:30:26.143493 30795 leader_election.cc:288] T 10becadd92c14019928d65c704724798 P 39b2209b30574a95bf503e074a4f7d29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cd339e18e549a9b1b07935fe4dd0fc (127.22.53.194:38789), 6d6bb899bd634ecd9fb3952a365d22ad (127.22.53.195:34909)
I20260926 07:30:26.148509 30783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:26.144181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.193:58289 (local address 127.22.53.195:34909)
0926 07:30:26.144305 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:26.144311 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:26.144327 (+ 16us) server_negotiation.cc:407] Connection header received
0926 07:30:26.144394 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:26.144396 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:26.144463 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:26.144563 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:30:26.145283 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:26.146024 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:26.147466 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:26.147885 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:26.148046 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:26.148108 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:26.148293 (+ 185us) server_negotiation.cc:299] Negotiation successful
0926 07:30:26.148378 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 07:30:26.148576 30861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:26.143774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.195:34909 (local address 127.22.53.193:58289)
0926 07:30:26.144145 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:26.144161 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:26.144272 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:26.144580 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:26.144583 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:26.144600 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:30:26.145139 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:26.145148 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:26.146165 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:26.146168 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:26.147327 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:26.147336 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:26.148047 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:26.148050 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:26.148153 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:26.148183 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 07:30:26.148275 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 07:30:26.148360 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"mutex_wait_us":1,"thread_start_us":78,"threads_started":1}
I20260926 07:30:26.148782 30787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:26.143891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.193:51697 (local address 127.22.53.194:38789)
0926 07:30:26.144047 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:26.144053 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:26.144470 (+ 417us) server_negotiation.cc:407] Connection header received
0926 07:30:26.144647 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:26.144651 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:26.144697 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:26.144757 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:30:26.145139 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:26.145691 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:26.147383 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:26.147995 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:26.148126 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:26.148177 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:26.148514 (+ 337us) server_negotiation.cc:299] Negotiation successful
0926 07:30:26.148649 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 07:30:26.149188 30714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10becadd92c14019928d65c704724798" candidate_uuid: "39b2209b30574a95bf503e074a4f7d29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d6bb899bd634ecd9fb3952a365d22ad" is_pre_election: true
I20260926 07:30:26.149354 30714 raft_consensus.cc:2454] T 10becadd92c14019928d65c704724798 P 6d6bb899bd634ecd9fb3952a365d22ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39b2209b30574a95bf503e074a4f7d29 in term 0.
I20260926 07:30:26.149626 30862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:26.143779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.194:38789 (local address 127.22.53.193:51697)
0926 07:30:26.144354 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:26.144368 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:26.144488 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:26.144887 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:26.144890 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:26.144906 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:30:26.145013 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:26.145019 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:26.145819 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:26.145824 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:26.147218 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:26.147230 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:26.148243 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:26.148247 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:26.148363 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:26.148388 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 07:30:26.149353 (+ 965us) client_negotiation.cc:240] Negotiation successful
0926 07:30:26.149473 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"mutex_wait_us":171,"thread_start_us":131,"threads_started":1}
I20260926 07:30:26.149786 30365 leader_election.cc:302] T 10becadd92c14019928d65c704724798 P 39b2209b30574a95bf503e074a4f7d29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39b2209b30574a95bf503e074a4f7d29, 6d6bb899bd634ecd9fb3952a365d22ad; no voters:
I20260926 07:30:26.150103 30795 raft_consensus.cc:2781] T 10becadd92c14019928d65c704724798 P 39b2209b30574a95bf503e074a4f7d29 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:30:26.150171 30795 raft_consensus.cc:482] T 10becadd92c14019928d65c704724798 P 39b2209b30574a95bf503e074a4f7d29 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:30:26.150209 30795 raft_consensus.cc:3037] T 10becadd92c14019928d65c704724798 P 39b2209b30574a95bf503e074a4f7d29 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:30:26.150345 30574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10becadd92c14019928d65c704724798" candidate_uuid: "39b2209b30574a95bf503e074a4f7d29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cd339e18e549a9b1b07935fe4dd0fc" is_pre_election: true
I20260926 07:30:26.150513 30574 raft_consensus.cc:2454] T 10becadd92c14019928d65c704724798 P d3cd339e18e549a9b1b07935fe4dd0fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39b2209b30574a95bf503e074a4f7d29 in term 0.
I20260926 07:30:26.151584 30795 raft_consensus.cc:504] T 10becadd92c14019928d65c704724798 P 39b2209b30574a95bf503e074a4f7d29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39b2209b30574a95bf503e074a4f7d29" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 34137 } } peers { permanent_uuid: "d3cd339e18e549a9b1b07935fe4dd0fc" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 38789 } } peers { permanent_uuid: "6d6bb899bd634ecd9fb3952a365d22ad" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 34909 } }
I20260926 07:30:26.151994 30795 leader_election.cc:288] T 10becadd92c14019928d65c704724798 P 39b2209b30574a95bf503e074a4f7d29 [CANDIDATE]: Term 1 election: Requested vote from peers d3cd339e18e549a9b1b07935fe4dd0fc (127.22.53.194:38789), 6d6bb899bd634ecd9fb3952a365d22ad (127.22.53.195:34909)
I20260926 07:30:26.152351 30574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10becadd92c14019928d65c704724798" candidate_uuid: "39b2209b30574a95bf503e074a4f7d29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cd339e18e549a9b1b07935fe4dd0fc"
I20260926 07:30:26.152413 30714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10becadd92c14019928d65c704724798" candidate_uuid: "39b2209b30574a95bf503e074a4f7d29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d6bb899bd634ecd9fb3952a365d22ad"
I20260926 07:30:26.152565 30574 raft_consensus.cc:3037] T 10becadd92c14019928d65c704724798 P d3cd339e18e549a9b1b07935fe4dd0fc [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:30:26.152570 30714 raft_consensus.cc:3037] T 10becadd92c14019928d65c704724798 P 6d6bb899bd634ecd9fb3952a365d22ad [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:30:26.153518 30574 raft_consensus.cc:2454] T 10becadd92c14019928d65c704724798 P d3cd339e18e549a9b1b07935fe4dd0fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39b2209b30574a95bf503e074a4f7d29 in term 1.
I20260926 07:30:26.153811 30714 raft_consensus.cc:2454] T 10becadd92c14019928d65c704724798 P 6d6bb899bd634ecd9fb3952a365d22ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39b2209b30574a95bf503e074a4f7d29 in term 1.
I20260926 07:30:26.153869 30366 leader_election.cc:302] T 10becadd92c14019928d65c704724798 P 39b2209b30574a95bf503e074a4f7d29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39b2209b30574a95bf503e074a4f7d29, d3cd339e18e549a9b1b07935fe4dd0fc; no voters:
I20260926 07:30:26.154083 30795 raft_consensus.cc:2781] T 10becadd92c14019928d65c704724798 P 39b2209b30574a95bf503e074a4f7d29 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:30:26.154316 30795 raft_consensus.cc:686] T 10becadd92c14019928d65c704724798 P 39b2209b30574a95bf503e074a4f7d29 [term 1 LEADER]: Becoming Leader. State: Replica: 39b2209b30574a95bf503e074a4f7d29, State: Running, Role: LEADER
I20260926 07:30:26.154593 30795 consensus_queue.cc:237] T 10becadd92c14019928d65c704724798 P 39b2209b30574a95bf503e074a4f7d29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39b2209b30574a95bf503e074a4f7d29" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 34137 } } peers { permanent_uuid: "d3cd339e18e549a9b1b07935fe4dd0fc" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 38789 } } peers { permanent_uuid: "6d6bb899bd634ecd9fb3952a365d22ad" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 34909 } }
I20260926 07:30:26.157465 30276 catalog_manager.cc:5495] T 10becadd92c14019928d65c704724798 P 39b2209b30574a95bf503e074a4f7d29 reported cstate change: term changed from 0 to 1, leader changed from <none> to 39b2209b30574a95bf503e074a4f7d29 (127.22.53.193). New cstate: current_term: 1 leader_uuid: "39b2209b30574a95bf503e074a4f7d29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39b2209b30574a95bf503e074a4f7d29" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 34137 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3cd339e18e549a9b1b07935fe4dd0fc" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 38789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d6bb899bd634ecd9fb3952a365d22ad" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 34909 } health_report { overall_health: UNKNOWN } } }
I20260926 07:30:26.171087 30785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:26.166320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42472 (local address 127.22.53.193:34137)
0926 07:30:26.166499 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:26.166512 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:26.166532 (+ 20us) server_negotiation.cc:407] Connection header received
0926 07:30:26.166793 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:26.166796 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:26.166874 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:26.167010 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:30:26.167275 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:26.167841 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:26.168899 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:26.169150 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:26.169279 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:26.169593 (+ 314us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:30:26.170187 (+ 594us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:30:26.170309 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:26.170850 (+ 541us) server_negotiation.cc:299] Negotiation successful
0926 07:30:26.170930 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 07:30:26.174597 30714 raft_consensus.cc:1261] T 10becadd92c14019928d65c704724798 P 6d6bb899bd634ecd9fb3952a365d22ad [term 1 FOLLOWER]: Refusing update from remote peer 39b2209b30574a95bf503e074a4f7d29: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:30:26.175032 30574 raft_consensus.cc:1261] T 10becadd92c14019928d65c704724798 P d3cd339e18e549a9b1b07935fe4dd0fc [term 1 FOLLOWER]: Refusing update from remote peer 39b2209b30574a95bf503e074a4f7d29: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:30:26.175828 30795 consensus_queue.cc:1035] T 10becadd92c14019928d65c704724798 P 39b2209b30574a95bf503e074a4f7d29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3cd339e18e549a9b1b07935fe4dd0fc" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 38789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:30:26.176429 30866 consensus_queue.cc:1035] T 10becadd92c14019928d65c704724798 P 39b2209b30574a95bf503e074a4f7d29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d6bb899bd634ecd9fb3952a365d22ad" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 34909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:30:26.195212 30273 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40530:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:30:26.195412 30273 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40530:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:30:26.197960 30273 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 945929f517c847deaee9a27a48db5aa9: Sending DeleteTablet for 3 replicas of tablet 10becadd92c14019928d65c704724798
I20260926 07:30:26.198680 30414 tablet_service.cc:1514] Processing DeleteTablet for tablet 10becadd92c14019928d65c704724798 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:30:26 UTC) from {username='oryx', principal='oryx/127.22.53.254@KRBTEST.COM'} at 127.0.0.1:42458
I20260926 07:30:26.198923 30845 tablet_replica.cc:330] T 10becadd92c14019928d65c704724798 P 39b2209b30574a95bf503e074a4f7d29: stopping tablet replica
I20260926 07:30:26.199102 30845 raft_consensus.cc:2229] T 10becadd92c14019928d65c704724798 P 39b2209b30574a95bf503e074a4f7d29 [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:30:26.199335 30553 tablet_service.cc:1514] Processing DeleteTablet for tablet 10becadd92c14019928d65c704724798 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:30:26 UTC) from {username='oryx', principal='oryx/127.22.53.254@KRBTEST.COM'} at 127.0.0.1:36126
I20260926 07:30:26.199476 30845 raft_consensus.cc:2258] T 10becadd92c14019928d65c704724798 P 39b2209b30574a95bf503e074a4f7d29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:30:26.199576 30844 tablet_replica.cc:330] T 10becadd92c14019928d65c704724798 P d3cd339e18e549a9b1b07935fe4dd0fc: stopping tablet replica
I20260926 07:30:26.199702 30844 raft_consensus.cc:2229] T 10becadd92c14019928d65c704724798 P d3cd339e18e549a9b1b07935fe4dd0fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:30:26.199827 30844 raft_consensus.cc:2258] T 10becadd92c14019928d65c704724798 P d3cd339e18e549a9b1b07935fe4dd0fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:30:26.200363 30694 tablet_service.cc:1514] Processing DeleteTablet for tablet 10becadd92c14019928d65c704724798 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:30:26 UTC) from {username='oryx', principal='oryx/127.22.53.254@KRBTEST.COM'} at 127.0.0.1:59978
I20260926 07:30:26.200605 30812 tablet_replica.cc:330] T 10becadd92c14019928d65c704724798 P 6d6bb899bd634ecd9fb3952a365d22ad: stopping tablet replica
I20260926 07:30:26.200732 30812 raft_consensus.cc:2229] T 10becadd92c14019928d65c704724798 P 6d6bb899bd634ecd9fb3952a365d22ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:30:26.200814 30844 ts_tablet_manager.cc:1905] T 10becadd92c14019928d65c704724798 P d3cd339e18e549a9b1b07935fe4dd0fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:30:26.200876 30812 raft_consensus.cc:2258] T 10becadd92c14019928d65c704724798 P 6d6bb899bd634ecd9fb3952a365d22ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:30:26.201067 30276 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40530:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:30:26.201551 30276 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:30:26.203744 30844 ts_tablet_manager.cc:1918] T 10becadd92c14019928d65c704724798 P d3cd339e18e549a9b1b07935fe4dd0fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:30:26.203846 30844 log.cc:1196] T 10becadd92c14019928d65c704724798 P d3cd339e18e549a9b1b07935fe4dd0fc: Deleting WAL directory at /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407783529717-22743-0/minicluster-data/ts-1/wal/wals/10becadd92c14019928d65c704724798
I20260926 07:30:26.204165 30844 ts_tablet_manager.cc:1939] T 10becadd92c14019928d65c704724798 P d3cd339e18e549a9b1b07935fe4dd0fc: Deleting consensus metadata
I20260926 07:30:26.205276 30262 catalog_manager.cc:4841] TS d3cd339e18e549a9b1b07935fe4dd0fc (127.22.53.194:38789): tablet 10becadd92c14019928d65c704724798 (table test-table [id=8a89c828871546d68b12432352b1ed8c]) successfully deleted
I20260926 07:30:26.208123 30845 ts_tablet_manager.cc:1905] T 10becadd92c14019928d65c704724798 P 39b2209b30574a95bf503e074a4f7d29: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:30:26.208173 30812 ts_tablet_manager.cc:1905] T 10becadd92c14019928d65c704724798 P 6d6bb899bd634ecd9fb3952a365d22ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:30:26.210806 30812 ts_tablet_manager.cc:1918] T 10becadd92c14019928d65c704724798 P 6d6bb899bd634ecd9fb3952a365d22ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:30:26.210909 30812 log.cc:1196] T 10becadd92c14019928d65c704724798 P 6d6bb899bd634ecd9fb3952a365d22ad: Deleting WAL directory at /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407783529717-22743-0/minicluster-data/ts-2/wal/wals/10becadd92c14019928d65c704724798
I20260926 07:30:26.211170 30812 ts_tablet_manager.cc:1939] T 10becadd92c14019928d65c704724798 P 6d6bb899bd634ecd9fb3952a365d22ad: Deleting consensus metadata
I20260926 07:30:26.211143 30845 ts_tablet_manager.cc:1918] T 10becadd92c14019928d65c704724798 P 39b2209b30574a95bf503e074a4f7d29: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:30:26.211233 30845 log.cc:1196] T 10becadd92c14019928d65c704724798 P 39b2209b30574a95bf503e074a4f7d29: Deleting WAL directory at /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407783529717-22743-0/minicluster-data/ts-0/wal/wals/10becadd92c14019928d65c704724798
I20260926 07:30:26.211511 30845 ts_tablet_manager.cc:1939] T 10becadd92c14019928d65c704724798 P 39b2209b30574a95bf503e074a4f7d29: Deleting consensus metadata
I20260926 07:30:26.213301 30553 tablet_service.cc:1467] Processing CreateTablet for tablet e00eb5cecfa44150afb53d6f655bec94 (DEFAULT_TABLE table=test-table [id=eb52edc718044c1c9cd039a45bbf4e0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:30:26.213515 30414 tablet_service.cc:1467] Processing CreateTablet for tablet e00eb5cecfa44150afb53d6f655bec94 (DEFAULT_TABLE table=test-table [id=eb52edc718044c1c9cd039a45bbf4e0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:30:26.213662 30553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e00eb5cecfa44150afb53d6f655bec94. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:30:26.213788 30414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e00eb5cecfa44150afb53d6f655bec94. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:30:26.216291 30788 tablet_bootstrap.cc:492] T e00eb5cecfa44150afb53d6f655bec94 P 39b2209b30574a95bf503e074a4f7d29: Bootstrap starting.
I20260926 07:30:26.216645 30790 tablet_bootstrap.cc:492] T e00eb5cecfa44150afb53d6f655bec94 P d3cd339e18e549a9b1b07935fe4dd0fc: Bootstrap starting.
I20260926 07:30:26.217288 30788 tablet_bootstrap.cc:654] T e00eb5cecfa44150afb53d6f655bec94 P 39b2209b30574a95bf503e074a4f7d29: Neither blocks nor log segments found. Creating new log.
I20260926 07:30:26.217470 30790 tablet_bootstrap.cc:654] T e00eb5cecfa44150afb53d6f655bec94 P d3cd339e18e549a9b1b07935fe4dd0fc: Neither blocks nor log segments found. Creating new log.
I20260926 07:30:26.212479 30263 catalog_manager.cc:4841] TS 39b2209b30574a95bf503e074a4f7d29 (127.22.53.193:34137): tablet 10becadd92c14019928d65c704724798 (table test-table [id=8a89c828871546d68b12432352b1ed8c]) successfully deleted
I20260926 07:30:26.218230 30261 catalog_manager.cc:4841] TS 6d6bb899bd634ecd9fb3952a365d22ad (127.22.53.195:34909): tablet 10becadd92c14019928d65c704724798 (table test-table [id=8a89c828871546d68b12432352b1ed8c]) successfully deleted
I20260926 07:30:26.218794 30694 tablet_service.cc:1467] Processing CreateTablet for tablet e00eb5cecfa44150afb53d6f655bec94 (DEFAULT_TABLE table=test-table [id=eb52edc718044c1c9cd039a45bbf4e0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:30:26.219080 30694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e00eb5cecfa44150afb53d6f655bec94. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:30:26.219287 30788 tablet_bootstrap.cc:492] T e00eb5cecfa44150afb53d6f655bec94 P 39b2209b30574a95bf503e074a4f7d29: No bootstrap required, opened a new log
I20260926 07:30:26.219399 30788 ts_tablet_manager.cc:1397] T e00eb5cecfa44150afb53d6f655bec94 P 39b2209b30574a95bf503e074a4f7d29: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260926 07:30:26.219899 30788 raft_consensus.cc:348] T e00eb5cecfa44150afb53d6f655bec94 P 39b2209b30574a95bf503e074a4f7d29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39b2209b30574a95bf503e074a4f7d29" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 34137 } } peers { permanent_uuid: "d3cd339e18e549a9b1b07935fe4dd0fc" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 38789 } } peers { permanent_uuid: "6d6bb899bd634ecd9fb3952a365d22ad" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 34909 } }
I20260926 07:30:26.220036 30788 raft_consensus.cc:374] T e00eb5cecfa44150afb53d6f655bec94 P 39b2209b30574a95bf503e074a4f7d29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:30:26.220077 30788 raft_consensus.cc:729] T e00eb5cecfa44150afb53d6f655bec94 P 39b2209b30574a95bf503e074a4f7d29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39b2209b30574a95bf503e074a4f7d29, State: Initialized, Role: FOLLOWER
I20260926 07:30:26.220286 30788 consensus_queue.cc:260] T e00eb5cecfa44150afb53d6f655bec94 P 39b2209b30574a95bf503e074a4f7d29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39b2209b30574a95bf503e074a4f7d29" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 34137 } } peers { permanent_uuid: "d3cd339e18e549a9b1b07935fe4dd0fc" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 38789 } } peers { permanent_uuid: "6d6bb899bd634ecd9fb3952a365d22ad" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 34909 } }
I20260926 07:30:26.220979 30788 ts_tablet_manager.cc:1428] T e00eb5cecfa44150afb53d6f655bec94 P 39b2209b30574a95bf503e074a4f7d29: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260926 07:30:26.222708 30790 tablet_bootstrap.cc:492] T e00eb5cecfa44150afb53d6f655bec94 P d3cd339e18e549a9b1b07935fe4dd0fc: No bootstrap required, opened a new log
I20260926 07:30:26.222777 30789 tablet_bootstrap.cc:492] T e00eb5cecfa44150afb53d6f655bec94 P 6d6bb899bd634ecd9fb3952a365d22ad: Bootstrap starting.
I20260926 07:30:26.222841 30790 ts_tablet_manager.cc:1397] T e00eb5cecfa44150afb53d6f655bec94 P d3cd339e18e549a9b1b07935fe4dd0fc: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260926 07:30:26.223568 30790 raft_consensus.cc:348] T e00eb5cecfa44150afb53d6f655bec94 P d3cd339e18e549a9b1b07935fe4dd0fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39b2209b30574a95bf503e074a4f7d29" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 34137 } } peers { permanent_uuid: "d3cd339e18e549a9b1b07935fe4dd0fc" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 38789 } } peers { permanent_uuid: "6d6bb899bd634ecd9fb3952a365d22ad" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 34909 } }
I20260926 07:30:26.223760 30790 raft_consensus.cc:374] T e00eb5cecfa44150afb53d6f655bec94 P d3cd339e18e549a9b1b07935fe4dd0fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:30:26.223860 30790 raft_consensus.cc:729] T e00eb5cecfa44150afb53d6f655bec94 P d3cd339e18e549a9b1b07935fe4dd0fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cd339e18e549a9b1b07935fe4dd0fc, State: Initialized, Role: FOLLOWER
I20260926 07:30:26.223939 30789 tablet_bootstrap.cc:654] T e00eb5cecfa44150afb53d6f655bec94 P 6d6bb899bd634ecd9fb3952a365d22ad: Neither blocks nor log segments found. Creating new log.
I20260926 07:30:26.224148 30790 consensus_queue.cc:260] T e00eb5cecfa44150afb53d6f655bec94 P d3cd339e18e549a9b1b07935fe4dd0fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39b2209b30574a95bf503e074a4f7d29" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 34137 } } peers { permanent_uuid: "d3cd339e18e549a9b1b07935fe4dd0fc" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 38789 } } peers { permanent_uuid: "6d6bb899bd634ecd9fb3952a365d22ad" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 34909 } }
I20260926 07:30:26.225785 30790 ts_tablet_manager.cc:1428] T e00eb5cecfa44150afb53d6f655bec94 P d3cd339e18e549a9b1b07935fe4dd0fc: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260926 07:30:26.230302 30789 tablet_bootstrap.cc:492] T e00eb5cecfa44150afb53d6f655bec94 P 6d6bb899bd634ecd9fb3952a365d22ad: No bootstrap required, opened a new log
I20260926 07:30:26.230458 30789 ts_tablet_manager.cc:1397] T e00eb5cecfa44150afb53d6f655bec94 P 6d6bb899bd634ecd9fb3952a365d22ad: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260926 07:30:26.230990 30789 raft_consensus.cc:348] T e00eb5cecfa44150afb53d6f655bec94 P 6d6bb899bd634ecd9fb3952a365d22ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39b2209b30574a95bf503e074a4f7d29" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 34137 } } peers { permanent_uuid: "d3cd339e18e549a9b1b07935fe4dd0fc" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 38789 } } peers { permanent_uuid: "6d6bb899bd634ecd9fb3952a365d22ad" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 34909 } }
I20260926 07:30:26.231137 30789 raft_consensus.cc:374] T e00eb5cecfa44150afb53d6f655bec94 P 6d6bb899bd634ecd9fb3952a365d22ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:30:26.231182 30789 raft_consensus.cc:729] T e00eb5cecfa44150afb53d6f655bec94 P 6d6bb899bd634ecd9fb3952a365d22ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d6bb899bd634ecd9fb3952a365d22ad, State: Initialized, Role: FOLLOWER
I20260926 07:30:26.231335 30789 consensus_queue.cc:260] T e00eb5cecfa44150afb53d6f655bec94 P 6d6bb899bd634ecd9fb3952a365d22ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39b2209b30574a95bf503e074a4f7d29" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 34137 } } peers { permanent_uuid: "d3cd339e18e549a9b1b07935fe4dd0fc" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 38789 } } peers { permanent_uuid: "6d6bb899bd634ecd9fb3952a365d22ad" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 34909 } }
I20260926 07:30:26.231667 30789 ts_tablet_manager.cc:1428] T e00eb5cecfa44150afb53d6f655bec94 P 6d6bb899bd634ecd9fb3952a365d22ad: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 07:30:26.367827 30819 raft_consensus.cc:482] T e00eb5cecfa44150afb53d6f655bec94 P d3cd339e18e549a9b1b07935fe4dd0fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:30:26.367995 30819 raft_consensus.cc:504] T e00eb5cecfa44150afb53d6f655bec94 P d3cd339e18e549a9b1b07935fe4dd0fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39b2209b30574a95bf503e074a4f7d29" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 34137 } } peers { permanent_uuid: "d3cd339e18e549a9b1b07935fe4dd0fc" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 38789 } } peers { permanent_uuid: "6d6bb899bd634ecd9fb3952a365d22ad" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 34909 } }
I20260926 07:30:26.368450 30819 leader_election.cc:288] T e00eb5cecfa44150afb53d6f655bec94 P d3cd339e18e549a9b1b07935fe4dd0fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39b2209b30574a95bf503e074a4f7d29 (127.22.53.193:34137), 6d6bb899bd634ecd9fb3952a365d22ad (127.22.53.195:34909)
I20260926 07:30:26.369007 30714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e00eb5cecfa44150afb53d6f655bec94" candidate_uuid: "d3cd339e18e549a9b1b07935fe4dd0fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d6bb899bd634ecd9fb3952a365d22ad" is_pre_election: true
I20260926 07:30:26.369006 30434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e00eb5cecfa44150afb53d6f655bec94" candidate_uuid: "d3cd339e18e549a9b1b07935fe4dd0fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39b2209b30574a95bf503e074a4f7d29" is_pre_election: true
I20260926 07:30:26.369155 30714 raft_consensus.cc:2454] T e00eb5cecfa44150afb53d6f655bec94 P 6d6bb899bd634ecd9fb3952a365d22ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3cd339e18e549a9b1b07935fe4dd0fc in term 0.
I20260926 07:30:26.369194 30434 raft_consensus.cc:2454] T e00eb5cecfa44150afb53d6f655bec94 P 39b2209b30574a95bf503e074a4f7d29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3cd339e18e549a9b1b07935fe4dd0fc in term 0.
I20260926 07:30:26.369587 30505 leader_election.cc:302] T e00eb5cecfa44150afb53d6f655bec94 P d3cd339e18e549a9b1b07935fe4dd0fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 39b2209b30574a95bf503e074a4f7d29, 6d6bb899bd634ecd9fb3952a365d22ad, d3cd339e18e549a9b1b07935fe4dd0fc; no voters:
I20260926 07:30:26.369812 30819 raft_consensus.cc:2781] T e00eb5cecfa44150afb53d6f655bec94 P d3cd339e18e549a9b1b07935fe4dd0fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:30:26.369889 30819 raft_consensus.cc:482] T e00eb5cecfa44150afb53d6f655bec94 P d3cd339e18e549a9b1b07935fe4dd0fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:30:26.369961 30819 raft_consensus.cc:3037] T e00eb5cecfa44150afb53d6f655bec94 P d3cd339e18e549a9b1b07935fe4dd0fc [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:30:26.371097 30819 raft_consensus.cc:504] T e00eb5cecfa44150afb53d6f655bec94 P d3cd339e18e549a9b1b07935fe4dd0fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39b2209b30574a95bf503e074a4f7d29" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 34137 } } peers { permanent_uuid: "d3cd339e18e549a9b1b07935fe4dd0fc" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 38789 } } peers { permanent_uuid: "6d6bb899bd634ecd9fb3952a365d22ad" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 34909 } }
I20260926 07:30:26.371542 30819 leader_election.cc:288] T e00eb5cecfa44150afb53d6f655bec94 P d3cd339e18e549a9b1b07935fe4dd0fc [CANDIDATE]: Term 1 election: Requested vote from peers 39b2209b30574a95bf503e074a4f7d29 (127.22.53.193:34137), 6d6bb899bd634ecd9fb3952a365d22ad (127.22.53.195:34909)
I20260926 07:30:26.371999 30714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e00eb5cecfa44150afb53d6f655bec94" candidate_uuid: "d3cd339e18e549a9b1b07935fe4dd0fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d6bb899bd634ecd9fb3952a365d22ad"
I20260926 07:30:26.372008 30434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e00eb5cecfa44150afb53d6f655bec94" candidate_uuid: "d3cd339e18e549a9b1b07935fe4dd0fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39b2209b30574a95bf503e074a4f7d29"
I20260926 07:30:26.372164 30714 raft_consensus.cc:3037] T e00eb5cecfa44150afb53d6f655bec94 P 6d6bb899bd634ecd9fb3952a365d22ad [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:30:26.372177 30434 raft_consensus.cc:3037] T e00eb5cecfa44150afb53d6f655bec94 P 39b2209b30574a95bf503e074a4f7d29 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:30:26.373335 30434 raft_consensus.cc:2454] T e00eb5cecfa44150afb53d6f655bec94 P 39b2209b30574a95bf503e074a4f7d29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3cd339e18e549a9b1b07935fe4dd0fc in term 1.
I20260926 07:30:26.373340 30714 raft_consensus.cc:2454] T e00eb5cecfa44150afb53d6f655bec94 P 6d6bb899bd634ecd9fb3952a365d22ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3cd339e18e549a9b1b07935fe4dd0fc in term 1.
I20260926 07:30:26.373740 30507 leader_election.cc:302] T e00eb5cecfa44150afb53d6f655bec94 P d3cd339e18e549a9b1b07935fe4dd0fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39b2209b30574a95bf503e074a4f7d29, d3cd339e18e549a9b1b07935fe4dd0fc; no voters:
I20260926 07:30:26.373966 30819 raft_consensus.cc:2781] T e00eb5cecfa44150afb53d6f655bec94 P d3cd339e18e549a9b1b07935fe4dd0fc [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:30:26.374087 30819 raft_consensus.cc:686] T e00eb5cecfa44150afb53d6f655bec94 P d3cd339e18e549a9b1b07935fe4dd0fc [term 1 LEADER]: Becoming Leader. State: Replica: d3cd339e18e549a9b1b07935fe4dd0fc, State: Running, Role: LEADER
I20260926 07:30:26.374250 30819 consensus_queue.cc:237] T e00eb5cecfa44150afb53d6f655bec94 P d3cd339e18e549a9b1b07935fe4dd0fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39b2209b30574a95bf503e074a4f7d29" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 34137 } } peers { permanent_uuid: "d3cd339e18e549a9b1b07935fe4dd0fc" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 38789 } } peers { permanent_uuid: "6d6bb899bd634ecd9fb3952a365d22ad" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 34909 } }
I20260926 07:30:26.376272 30276 catalog_manager.cc:5495] T e00eb5cecfa44150afb53d6f655bec94 P d3cd339e18e549a9b1b07935fe4dd0fc reported cstate change: term changed from 0 to 1, leader changed from <none> to d3cd339e18e549a9b1b07935fe4dd0fc (127.22.53.194). New cstate: current_term: 1 leader_uuid: "d3cd339e18e549a9b1b07935fe4dd0fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39b2209b30574a95bf503e074a4f7d29" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 34137 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3cd339e18e549a9b1b07935fe4dd0fc" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 38789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6d6bb899bd634ecd9fb3952a365d22ad" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 34909 } health_report { overall_health: UNKNOWN } } }
I20260926 07:30:26.400166 30787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:26.396918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36150 (local address 127.22.53.194:38789)
0926 07:30:26.397052 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:26.397057 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:26.397071 (+ 14us) server_negotiation.cc:407] Connection header received
0926 07:30:26.397203 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:26.397207 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:26.397264 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:26.397357 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:30:26.397599 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:26.398124 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:26.398997 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:26.399258 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:26.399386 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:26.399645 (+ 259us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:30:26.399738 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:30:26.399824 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:26.399937 (+ 113us) server_negotiation.cc:299] Negotiation successful
0926 07:30:26.399998 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 07:30:26.408048 30434 raft_consensus.cc:1261] T e00eb5cecfa44150afb53d6f655bec94 P 39b2209b30574a95bf503e074a4f7d29 [term 1 FOLLOWER]: Refusing update from remote peer d3cd339e18e549a9b1b07935fe4dd0fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:30:26.408119 30714 raft_consensus.cc:1261] T e00eb5cecfa44150afb53d6f655bec94 P 6d6bb899bd634ecd9fb3952a365d22ad [term 1 FOLLOWER]: Refusing update from remote peer d3cd339e18e549a9b1b07935fe4dd0fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:30:26.408488 30836 consensus_queue.cc:1035] T e00eb5cecfa44150afb53d6f655bec94 P d3cd339e18e549a9b1b07935fe4dd0fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "39b2209b30574a95bf503e074a4f7d29" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 34137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:30:26.408993 30836 consensus_queue.cc:1035] T e00eb5cecfa44150afb53d6f655bec94 P d3cd339e18e549a9b1b07935fe4dd0fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d6bb899bd634ecd9fb3952a365d22ad" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 34909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:30:26.448215 30276 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40530:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:30:26.448410 30276 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40530:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:30:26.450892 30276 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 945929f517c847deaee9a27a48db5aa9: Sending DeleteTablet for 3 replicas of tablet e00eb5cecfa44150afb53d6f655bec94
I20260926 07:30:26.451655 30414 tablet_service.cc:1514] Processing DeleteTablet for tablet e00eb5cecfa44150afb53d6f655bec94 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:30:26 UTC) from {username='oryx', principal='oryx/127.22.53.254@KRBTEST.COM'} at 127.0.0.1:42458
I20260926 07:30:26.451704 30694 tablet_service.cc:1514] Processing DeleteTablet for tablet e00eb5cecfa44150afb53d6f655bec94 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:30:26 UTC) from {username='oryx', principal='oryx/127.22.53.254@KRBTEST.COM'} at 127.0.0.1:59978
I20260926 07:30:26.451895 30812 tablet_replica.cc:330] T e00eb5cecfa44150afb53d6f655bec94 P 6d6bb899bd634ecd9fb3952a365d22ad: stopping tablet replica
I20260926 07:30:26.451939 30845 tablet_replica.cc:330] T e00eb5cecfa44150afb53d6f655bec94 P 39b2209b30574a95bf503e074a4f7d29: stopping tablet replica
I20260926 07:30:26.452054 30812 raft_consensus.cc:2229] T e00eb5cecfa44150afb53d6f655bec94 P 6d6bb899bd634ecd9fb3952a365d22ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:30:26.452060 30554 tablet_service.cc:1514] Processing DeleteTablet for tablet e00eb5cecfa44150afb53d6f655bec94 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:30:26 UTC) from {username='oryx', principal='oryx/127.22.53.254@KRBTEST.COM'} at 127.0.0.1:36126
I20260926 07:30:26.452085 30845 raft_consensus.cc:2229] T e00eb5cecfa44150afb53d6f655bec94 P 39b2209b30574a95bf503e074a4f7d29 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:30:26.452195 30812 raft_consensus.cc:2258] T e00eb5cecfa44150afb53d6f655bec94 P 6d6bb899bd634ecd9fb3952a365d22ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:30:26.452216 30845 raft_consensus.cc:2258] T e00eb5cecfa44150afb53d6f655bec94 P 39b2209b30574a95bf503e074a4f7d29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:30:26.452239 30844 tablet_replica.cc:330] T e00eb5cecfa44150afb53d6f655bec94 P d3cd339e18e549a9b1b07935fe4dd0fc: stopping tablet replica
I20260926 07:30:26.452359 30844 raft_consensus.cc:2229] T e00eb5cecfa44150afb53d6f655bec94 P d3cd339e18e549a9b1b07935fe4dd0fc [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:30:26.452561 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 30349
I20260926 07:30:26.452607 30844 raft_consensus.cc:2258] T e00eb5cecfa44150afb53d6f655bec94 P d3cd339e18e549a9b1b07935fe4dd0fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:30:26.453152 30812 ts_tablet_manager.cc:1905] T e00eb5cecfa44150afb53d6f655bec94 P 6d6bb899bd634ecd9fb3952a365d22ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:30:26.454449 30844 ts_tablet_manager.cc:1905] T e00eb5cecfa44150afb53d6f655bec94 P d3cd339e18e549a9b1b07935fe4dd0fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:30:26.456063 30812 ts_tablet_manager.cc:1918] T e00eb5cecfa44150afb53d6f655bec94 P 6d6bb899bd634ecd9fb3952a365d22ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 07:30:26.456143 30812 log.cc:1196] T e00eb5cecfa44150afb53d6f655bec94 P 6d6bb899bd634ecd9fb3952a365d22ad: Deleting WAL directory at /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407783529717-22743-0/minicluster-data/ts-2/wal/wals/e00eb5cecfa44150afb53d6f655bec94
I20260926 07:30:26.456415 30812 ts_tablet_manager.cc:1939] T e00eb5cecfa44150afb53d6f655bec94 P 6d6bb899bd634ecd9fb3952a365d22ad: Deleting consensus metadata
I20260926 07:30:26.457839 30261 catalog_manager.cc:4841] TS 6d6bb899bd634ecd9fb3952a365d22ad (127.22.53.195:34909): tablet e00eb5cecfa44150afb53d6f655bec94 (table test-table [id=eb52edc718044c1c9cd039a45bbf4e0e]) successfully deleted
I20260926 07:30:26.457952 30844 ts_tablet_manager.cc:1918] T e00eb5cecfa44150afb53d6f655bec94 P d3cd339e18e549a9b1b07935fe4dd0fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 07:30:26.458069 30844 log.cc:1196] T e00eb5cecfa44150afb53d6f655bec94 P d3cd339e18e549a9b1b07935fe4dd0fc: Deleting WAL directory at /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407783529717-22743-0/minicluster-data/ts-1/wal/wals/e00eb5cecfa44150afb53d6f655bec94
I20260926 07:30:26.458340 30844 ts_tablet_manager.cc:1939] T e00eb5cecfa44150afb53d6f655bec94 P d3cd339e18e549a9b1b07935fe4dd0fc: Deleting consensus metadata
I20260926 07:30:26.459157 30262 catalog_manager.cc:4841] TS d3cd339e18e549a9b1b07935fe4dd0fc (127.22.53.194:38789): tablet e00eb5cecfa44150afb53d6f655bec94 (table test-table [id=eb52edc718044c1c9cd039a45bbf4e0e]) successfully deleted
W20260926 07:30:26.463666 30263 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.53.193:34137 (error 108)
I20260926 07:30:26.464569 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 30489
I20260926 07:30:26.465811 30877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:26.464851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.193:34137 (local address 127.0.0.1:42478)
0926 07:30:26.465550 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:26.465650 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.193:34137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":81,"threads_started":1}
W20260926 07:30:26.465956 30263 catalog_manager.cc:4584] TS 39b2209b30574a95bf503e074a4f7d29 (127.22.53.193:34137): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e00eb5cecfa44150afb53d6f655bec94: Network error: Client connection negotiation failed: client connection to 127.22.53.193:34137: connect: Connection refused (error 111)
I20260926 07:30:26.473973 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 30629
I20260926 07:30:26.483664 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 30245
2026-09-26T07:30:26Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3511 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790407783529717-22743-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:30:26 dist-test-slave-wdjc krb5kdc[30880](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:30:26 dist-test-slave-wdjc krb5kdc[30880](info): set up 2 sockets
Sep 26 07:30:26 dist-test-slave-wdjc krb5kdc[30880](info): commencing operation
krb5kdc: starting...
W20260926 07:30:27.569706 22743 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790407783529717-22743-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790407783529717-22743-0/krb5kdc/test-user.keytab.
Sep 26 07:30:27 dist-test-slave-wdjc krb5kdc[30880](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407827, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:30:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:30:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.53.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.53.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.53.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab.
Entry for principal kudu/127.22.53.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab.
I20260926 07:30:27.708604 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790407783529717-22743-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790407783529717-22743-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790407783529717-22743-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.53.254:42473
--webserver_interface=127.22.53.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab
--principal=kudu/127.22.53.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:39921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.53.254:42473
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:30:27.829885 30893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:30:27.830231 30893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:30:27.830277 30893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:30:27.834921 30893 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:30:27.835021 30893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:30:27.835043 30893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:30:27.835060 30893 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:30:27.835078 30893 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:30:27.841439 30893 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:39921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790407783529717-22743-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790407783529717-22743-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.53.254:42473
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.53.254:42473
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.53.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30893
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790407783529717-22743-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:30:27.842744 30893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:30:27.843906 30893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:30:27.853971 30901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:27.853971 30898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:27.854015 30899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:27.854460 30893 server_base.cc:1034] running on GCE node
I20260926 07:30:27.855156 30893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:30:27.856693 30893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:30:27.857932 30893 hybrid_clock.cc:648] HybridClock initialized: now 1790407827857915 us; error 39 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-26T07:30:27Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1375 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:30:27 dist-test-slave-wdjc krb5kdc[30906](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:30:27 dist-test-slave-wdjc krb5kdc[30906](info): set up 2 sockets
Sep 26 07:30:27 dist-test-slave-wdjc krb5kdc[30906](info): commencing operation
krb5kdc: starting...
W20260926 07:30:28.925709 22743 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/krb5kdc/test-user.keytab.
Sep 26 07:30:29 dist-test-slave-wdjc krb5kdc[30906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407829, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:30:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:30:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.53.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.53.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.53.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab.
Entry for principal kudu/127.22.53.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab.
I20260926 07:30:29.060184 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.53.254:36293
--webserver_interface=127.22.53.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab
--principal=kudu/127.22.53.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:45251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.53.254:36293
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:30:29.179929 30919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:30:29.180305 30919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:30:29.180387 30919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:30:29.184885 30919 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:30:29.185022 30919 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:30:29.185057 30919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:30:29.185096 30919 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:30:29.185132 30919 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:30:29.191190 30919 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:45251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.53.254:36293
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab
--principal=kudu/127.22.53.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.53.254:36293
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.53.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30919
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:30:29.192468 30919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:30:29.193665 30919 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:30:29.203287 30927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:29.203231 30925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:29.203231 30924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:29.203933 30919 server_base.cc:1034] running on GCE node
I20260926 07:30:29.204654 30919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:30:29.206007 30919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:30:29.207211 30919 hybrid_clock.cc:648] HybridClock initialized: now 1790407829207192 us; error 38 us; skew 500 ppm
Sep 26 07:30:29 dist-test-slave-wdjc krb5kdc[30906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407829, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:30:29.210410 30919 init.cc:377] Logged in from keytab as kudu/127.22.53.254@KRBTEST.COM (short username kudu)
I20260926 07:30:29.211503 30919 webserver.cc:460] Webserver started at http://127.22.53.254:38905/ using document root <none> and password file <none>
I20260926 07:30:29.212131 30919 fs_manager.cc:359] Metadata directory not provided
I20260926 07:30:29.212201 30919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:30:29.212447 30919 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:30:29.214787 30919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/master-0/data/instance:
uuid: "c96bea3d625746eeab51cbfc0670cf67"
format_stamp: "Formatted at 2026-09-26 07:30:29 on dist-test-slave-wdjc"
server_key: "b262d9ff4f4a14abf80661fe68829b9d"
server_key_iv: "40c90a474a2c0be7301cd5f1c68d84a2"
server_key_version: "encryptionkey@0"
I20260926 07:30:29.215440 30919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/master-0/wal/instance:
uuid: "c96bea3d625746eeab51cbfc0670cf67"
format_stamp: "Formatted at 2026-09-26 07:30:29 on dist-test-slave-wdjc"
server_key: "b262d9ff4f4a14abf80661fe68829b9d"
server_key_iv: "40c90a474a2c0be7301cd5f1c68d84a2"
server_key_version: "encryptionkey@0"
I20260926 07:30:29.220150 30919 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260926 07:30:29.223145 30934 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:29.224738 30919 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:30:29.224927 30919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/master-0/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/master-0/wal
uuid: "c96bea3d625746eeab51cbfc0670cf67"
format_stamp: "Formatted at 2026-09-26 07:30:29 on dist-test-slave-wdjc"
server_key: "b262d9ff4f4a14abf80661fe68829b9d"
server_key_iv: "40c90a474a2c0be7301cd5f1c68d84a2"
server_key_version: "encryptionkey@0"
I20260926 07:30:29.225073 30919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:30:29.255795 30919 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:30:29.259495 30919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:30:29.259742 30919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:30:29.267580 30919 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.254:36293
I20260926 07:30:29.267619 30986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.254:36293 every 8 connection(s)
I20260926 07:30:29.268726 30919 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
I20260926 07:30:29.271829 30987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:30:29.277963 30987 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c96bea3d625746eeab51cbfc0670cf67: Bootstrap starting.
I20260926 07:30:29.278189 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 30919
I20260926 07:30:29.278362 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/master-0/wal/instance
I20260926 07:30:29.278654 22743 external_mini_cluster.cc:1428] Setting key 9848f3d565603e81d22c4bd442a8b1b7
I20260926 07:30:29.280946 30987 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c96bea3d625746eeab51cbfc0670cf67: Neither blocks nor log segments found. Creating new log.
I20260926 07:30:29.281749 30987 log.cc:824] T 00000000000000000000000000000000 P c96bea3d625746eeab51cbfc0670cf67: Log is configured to *not* fsync() on all Append() calls
I20260926 07:30:29.283747 30987 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c96bea3d625746eeab51cbfc0670cf67: No bootstrap required, opened a new log
I20260926 07:30:29.287433 30987 raft_consensus.cc:348] T 00000000000000000000000000000000 P c96bea3d625746eeab51cbfc0670cf67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c96bea3d625746eeab51cbfc0670cf67" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 36293 } }
I20260926 07:30:29.287683 30987 raft_consensus.cc:374] T 00000000000000000000000000000000 P c96bea3d625746eeab51cbfc0670cf67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:30:29.287719 30987 raft_consensus.cc:729] T 00000000000000000000000000000000 P c96bea3d625746eeab51cbfc0670cf67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c96bea3d625746eeab51cbfc0670cf67, State: Initialized, Role: FOLLOWER
I20260926 07:30:29.288218 30987 consensus_queue.cc:260] T 00000000000000000000000000000000 P c96bea3d625746eeab51cbfc0670cf67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c96bea3d625746eeab51cbfc0670cf67" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 36293 } }
I20260926 07:30:29.288396 30987 raft_consensus.cc:388] T 00000000000000000000000000000000 P c96bea3d625746eeab51cbfc0670cf67 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:30:29.288486 30987 raft_consensus.cc:482] T 00000000000000000000000000000000 P c96bea3d625746eeab51cbfc0670cf67 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:30:29.288641 30987 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c96bea3d625746eeab51cbfc0670cf67 [term 0 FOLLOWER]: Advancing to term 1
Sep 26 07:30:29 dist-test-slave-wdjc krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407829, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.53.254@KRBTEST.COM
I20260926 07:30:29.289853 30987 raft_consensus.cc:504] T 00000000000000000000000000000000 P c96bea3d625746eeab51cbfc0670cf67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c96bea3d625746eeab51cbfc0670cf67" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 36293 } }
I20260926 07:30:29.290298 30987 leader_election.cc:302] T 00000000000000000000000000000000 P c96bea3d625746eeab51cbfc0670cf67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c96bea3d625746eeab51cbfc0670cf67; no voters:
I20260926 07:30:29.290759 30987 leader_election.cc:288] T 00000000000000000000000000000000 P c96bea3d625746eeab51cbfc0670cf67 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:30:29.290874 30992 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c96bea3d625746eeab51cbfc0670cf67 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:30:29.291544 30992 raft_consensus.cc:686] T 00000000000000000000000000000000 P c96bea3d625746eeab51cbfc0670cf67 [term 1 LEADER]: Becoming Leader. State: Replica: c96bea3d625746eeab51cbfc0670cf67, State: Running, Role: LEADER
I20260926 07:30:29.291934 30992 consensus_queue.cc:237] T 00000000000000000000000000000000 P c96bea3d625746eeab51cbfc0670cf67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c96bea3d625746eeab51cbfc0670cf67" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 36293 } }
I20260926 07:30:29.292297 30987 sys_catalog.cc:564] T 00000000000000000000000000000000 P c96bea3d625746eeab51cbfc0670cf67 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:30:29.293751 30994 sys_catalog.cc:455] T 00000000000000000000000000000000 P c96bea3d625746eeab51cbfc0670cf67 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c96bea3d625746eeab51cbfc0670cf67. Latest consensus state: current_term: 1 leader_uuid: "c96bea3d625746eeab51cbfc0670cf67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c96bea3d625746eeab51cbfc0670cf67" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 36293 } } }
I20260926 07:30:29.293890 30994 sys_catalog.cc:458] T 00000000000000000000000000000000 P c96bea3d625746eeab51cbfc0670cf67 [sys.catalog]: This master's current role is: LEADER
I20260926 07:30:29.294215 30993 sys_catalog.cc:455] T 00000000000000000000000000000000 P c96bea3d625746eeab51cbfc0670cf67 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c96bea3d625746eeab51cbfc0670cf67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c96bea3d625746eeab51cbfc0670cf67" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 36293 } } }
I20260926 07:30:29.294319 30993 sys_catalog.cc:458] T 00000000000000000000000000000000 P c96bea3d625746eeab51cbfc0670cf67 [sys.catalog]: This master's current role is: LEADER
I20260926 07:30:29.296236 30990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:29.280275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50292 (local address 127.22.53.254:36293)
0926 07:30:29.280785 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:29.280803 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:29.280841 (+ 38us) server_negotiation.cc:407] Connection header received
0926 07:30:29.281648 (+ 807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:29.281669 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:29.282119 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:29.282516 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:29.283419 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:29.284666 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:29.285637 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:29.286179 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:29.286381 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:29.289491 (+ 3110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:29.289501 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:29.289524 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:29.289562 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:29.292016 (+ 2454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:29.292692 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:29.292698 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:29.292702 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:29.292843 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:29.293206 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:29.293209 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:29.293211 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:29.293766 (+ 555us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:29.293973 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:29.294864 (+ 891us) server_negotiation.cc:299] Negotiation successful
0926 07:30:29.295226 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":161,"threads_started":1}
W20260926 07:30:29.298048 31007 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c96bea3d625746eeab51cbfc0670cf67: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:30:29.298137 31007 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:30:29.298220 31002 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:30:29.299165 31002 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:30:29.305967 31002 catalog_manager.cc:1347] Generated new cluster ID: 5d4899d213984bc2b7a265640ca2e072
I20260926 07:30:29.306066 31002 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:30:29.321905 31002 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:30:29.323338 31002 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:30:29.331704 31002 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c96bea3d625746eeab51cbfc0670cf67: Generated new TSK 0
I20260926 07:30:29.332726 31002 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.53.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.53.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.53.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.193.keytab.
Entry for principal kudu/127.22.53.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.193.keytab.
I20260926 07:30:29.363492 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.53.193:0
--local_ip_for_outbound_sockets=127.22.53.193
--webserver_interface=127.22.53.193
--webserver_port=0
--tserver_master_addrs=127.22.53.254:36293
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.193.keytab
--principal=kudu/127.22.53.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:45251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:30:29.484069 31013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:30:29.484421 31013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:30:29.484503 31013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:30:29.488669 31013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:30:29.488790 31013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:30:29.488890 31013 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.53.193
I20260926 07:30:29.494964 31013 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:45251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.193.keytab
--principal=kudu/127.22.53.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.53.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.53.193
--webserver_port=0
--tserver_master_addrs=127.22.53.254:36293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31013
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.53.193
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:30:29.496214 31013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:30:29.497370 31013 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:30:29.510397 31019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:29.510394 31018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:29.510411 31021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:29.510813 31013 server_base.cc:1034] running on GCE node
I20260926 07:30:29.511255 31013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:30:29.511996 31013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:30:29.513227 31013 hybrid_clock.cc:648] HybridClock initialized: now 1790407829513205 us; error 34 us; skew 500 ppm
Sep 26 07:30:29 dist-test-slave-wdjc krb5kdc[30906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407829, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:30:29.516637 31013 init.cc:377] Logged in from keytab as kudu/127.22.53.193@KRBTEST.COM (short username kudu)
I20260926 07:30:29.517858 31013 webserver.cc:460] Webserver started at http://127.22.53.193:45241/ using document root <none> and password file <none>
I20260926 07:30:29.518510 31013 fs_manager.cc:359] Metadata directory not provided
I20260926 07:30:29.518582 31013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:30:29.518826 31013 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:30:29.521191 31013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/ts-0/data/instance:
uuid: "f01aecc1b8644dd6b7398823c2df278b"
format_stamp: "Formatted at 2026-09-26 07:30:29 on dist-test-slave-wdjc"
server_key: "5d4943cc727c7b35baf469518e7e6a8c"
server_key_iv: "c48e2e45b8037e1f8e681cf8dff6710c"
server_key_version: "encryptionkey@0"
I20260926 07:30:29.521831 31013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/ts-0/wal/instance:
uuid: "f01aecc1b8644dd6b7398823c2df278b"
format_stamp: "Formatted at 2026-09-26 07:30:29 on dist-test-slave-wdjc"
server_key: "5d4943cc727c7b35baf469518e7e6a8c"
server_key_iv: "c48e2e45b8037e1f8e681cf8dff6710c"
server_key_version: "encryptionkey@0"
I20260926 07:30:29.526086 31013 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 07:30:29.529136 31028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:29.530624 31013 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:30:29.530793 31013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/ts-0/wal
uuid: "f01aecc1b8644dd6b7398823c2df278b"
format_stamp: "Formatted at 2026-09-26 07:30:29 on dist-test-slave-wdjc"
server_key: "5d4943cc727c7b35baf469518e7e6a8c"
server_key_iv: "c48e2e45b8037e1f8e681cf8dff6710c"
server_key_version: "encryptionkey@0"
I20260926 07:30:29.530930 31013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:30:29.546494 31013 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:30:29.550076 31013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:30:29.550315 31013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:30:29.551079 31013 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:30:29.552062 31013 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:30:29.552142 31013 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:29.552219 31013 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:30:29.552264 31013 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:29.563069 31013 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.193:44951
I20260926 07:30:29.563099 31141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.193:44951 every 8 connection(s)
I20260926 07:30:29.564221 31013 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/ts-0/data/info.pb
I20260926 07:30:29.571895 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 31013
I20260926 07:30:29.572059 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/ts-0/wal/instance
I20260926 07:30:29.572388 22743 external_mini_cluster.cc:1428] Setting key 776369e65856511f90de437ba45440a6
Sep 26 07:30:29 dist-test-slave-wdjc krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407829, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.193@KRBTEST.COM for kudu/127.22.53.254@KRBTEST.COM
I20260926 07:30:29.581308 30990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:29.566463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.193:36119 (local address 127.22.53.254:36293)
0926 07:30:29.566655 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:29.566660 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:29.567730 (+ 1070us) server_negotiation.cc:407] Connection header received
0926 07:30:29.569316 (+ 1586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:29.569319 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:29.569376 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:29.569462 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:29.570827 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:29.571477 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:29.573264 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:29.573608 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:29.573741 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:29.577064 (+ 3323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:29.577068 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:29.577070 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:29.577099 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:29.578797 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:29.579434 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:29.579438 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:29.579439 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:29.579521 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:29.579944 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:29.579948 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:29.579949 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:29.580162 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:29.580287 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:29.580952 (+ 665us) server_negotiation.cc:299] Negotiation successful
0926 07:30:29.581102 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":89}
I20260926 07:30:29.582196 31144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:29.566932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:36293 (local address 127.22.53.193:36119)
0926 07:30:29.567524 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:29.567578 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:29.568980 (+ 1402us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:29.569656 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:29.569665 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:29.570083 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:29.570617 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:29.570632 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:29.571688 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:29.571695 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:29.573116 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:29.573130 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:29.573798 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:29.573803 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:29.573984 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:29.574681 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:29.574706 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:29.576834 (+ 2128us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:30:29.578957 (+ 2123us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:29.578965 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:29.578985 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:29.579303 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:29.579628 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:29.579632 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:29.579634 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:29.579827 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:29.580281 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:30:29.580288 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:30:29.580668 (+ 380us) client_negotiation.cc:769] Sending connection context
0926 07:30:29.580892 (+ 224us) client_negotiation.cc:240] Negotiation successful
0926 07:30:29.581270 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":144,"threads_started":1}
WARNING: no policy specified for kudu/127.22.53.194@KRBTEST.COM; defaulting to no policy
I20260926 07:30:29.583995 31142 heartbeater.cc:347] Connected to a master server at 127.22.53.254:36293
I20260926 07:30:29.584270 31142 heartbeater.cc:464] Registering TS with master...
I20260926 07:30:29.584975 31142 heartbeater.cc:511] Master 127.22.53.254:36293 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.53.194@KRBTEST.COM" created.
I20260926 07:30:29.586901 30951 ts_manager.cc:194] Registered new tserver with Master: f01aecc1b8644dd6b7398823c2df278b (127.22.53.193:44951)
I20260926 07:30:29.588544 30951 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.53.193@KRBTEST.COM'} at 127.22.53.193:36119
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.53.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.194.keytab.
Entry for principal kudu/127.22.53.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.194.keytab.
I20260926 07:30:29.604398 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.53.194:0
--local_ip_for_outbound_sockets=127.22.53.194
--webserver_interface=127.22.53.194
--webserver_port=0
--tserver_master_addrs=127.22.53.254:36293
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.194.keytab
--principal=kudu/127.22.53.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:45251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:30:29.723778 31147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:30:29.724107 31147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:30:29.724153 31147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:30:29.728719 31147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:30:29.728816 31147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:30:29.728910 31147 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.53.194
I20260926 07:30:29.734887 31147 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:45251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.194.keytab
--principal=kudu/127.22.53.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.53.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.53.194
--webserver_port=0
--tserver_master_addrs=127.22.53.254:36293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31147
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.53.194
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:30:29.736132 31147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:30:29.737843 31147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:30:29.748569 31152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:29.748570 31153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:29.748576 31155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:29.748965 31147 server_base.cc:1034] running on GCE node
I20260926 07:30:29.749480 31147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:30:29.750236 31147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:30:29.751448 31147 hybrid_clock.cc:648] HybridClock initialized: now 1790407829751427 us; error 37 us; skew 500 ppm
Sep 26 07:30:29 dist-test-slave-wdjc krb5kdc[30906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407829, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:30:29.755021 31147 init.cc:377] Logged in from keytab as kudu/127.22.53.194@KRBTEST.COM (short username kudu)
I20260926 07:30:29.756156 31147 webserver.cc:460] Webserver started at http://127.22.53.194:45857/ using document root <none> and password file <none>
I20260926 07:30:29.756935 31147 fs_manager.cc:359] Metadata directory not provided
I20260926 07:30:29.757051 31147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:30:29.757321 31147 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:30:29.759783 31147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/ts-1/data/instance:
uuid: "cf15527bf98f4c168972d464e1635d84"
format_stamp: "Formatted at 2026-09-26 07:30:29 on dist-test-slave-wdjc"
server_key: "d58727c1760d61980c4035fad02d9d1a"
server_key_iv: "11ec095ce093578a1c9a2193ce90f0c1"
server_key_version: "encryptionkey@0"
I20260926 07:30:29.760489 31147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/ts-1/wal/instance:
uuid: "cf15527bf98f4c168972d464e1635d84"
format_stamp: "Formatted at 2026-09-26 07:30:29 on dist-test-slave-wdjc"
server_key: "d58727c1760d61980c4035fad02d9d1a"
server_key_iv: "11ec095ce093578a1c9a2193ce90f0c1"
server_key_version: "encryptionkey@0"
I20260926 07:30:29.765235 31147 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 07:30:29.768280 31162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:29.769891 31147 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:30:29.770092 31147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/ts-1/wal
uuid: "cf15527bf98f4c168972d464e1635d84"
format_stamp: "Formatted at 2026-09-26 07:30:29 on dist-test-slave-wdjc"
server_key: "d58727c1760d61980c4035fad02d9d1a"
server_key_iv: "11ec095ce093578a1c9a2193ce90f0c1"
server_key_version: "encryptionkey@0"
I20260926 07:30:29.770296 31147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:30:29.795778 31147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:30:29.799224 31147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:30:29.799458 31147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:30:29.800215 31147 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:30:29.801237 31147 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:30:29.801317 31147 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:29.801388 31147 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:30:29.801434 31147 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:29.811324 31147 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.194:42587
I20260926 07:30:29.811401 31275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.194:42587 every 8 connection(s)
I20260926 07:30:29.812435 31147 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/ts-1/data/info.pb
I20260926 07:30:29.822604 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 31147
I20260926 07:30:29.822799 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/ts-1/wal/instance
Sep 26 07:30:29 dist-test-slave-wdjc krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407829, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.194@KRBTEST.COM for kudu/127.22.53.254@KRBTEST.COM
I20260926 07:30:29.823169 22743 external_mini_cluster.cc:1428] Setting key ffad0deb5c274bb2266a1fd0fa07b730
I20260926 07:30:29.829044 30990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:29.814557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.194:58651 (local address 127.22.53.254:36293)
0926 07:30:29.814741 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:29.814747 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:29.815729 (+ 982us) server_negotiation.cc:407] Connection header received
0926 07:30:29.817143 (+ 1414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:29.817147 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:29.817201 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:29.817287 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:29.818647 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:29.819231 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:29.820446 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:29.820768 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:29.820870 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:29.823961 (+ 3091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:29.823966 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:29.823969 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:29.824005 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:29.826005 (+ 2000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:29.826652 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:29.826660 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:29.826663 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:29.826787 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:29.827369 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:29.827372 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:29.827373 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:29.827711 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:29.827846 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:29.828673 (+ 827us) server_negotiation.cc:299] Negotiation successful
0926 07:30:29.828871 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 07:30:29.830168 31278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:29.814993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:36293 (local address 127.22.53.194:58651)
0926 07:30:29.815548 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:29.815596 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:29.816883 (+ 1287us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:29.817510 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:29.817518 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:29.817945 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:29.818466 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:29.818481 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:29.819374 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:29.819377 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:29.820311 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:29.820321 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:29.820944 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:29.820947 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:29.821128 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:29.821807 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:29.821826 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:29.823663 (+ 1837us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:30:29.826180 (+ 2517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:29.826186 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:29.826211 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:29.826525 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:29.826941 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:29.826946 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:29.826950 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:29.827186 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:29.827920 (+ 734us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:30:29.827928 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:30:29.828328 (+ 400us) client_negotiation.cc:769] Sending connection context
0926 07:30:29.828618 (+ 290us) client_negotiation.cc:240] Negotiation successful
0926 07:30:29.829063 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":170,"threads_started":1}
I20260926 07:30:29.831986 31276 heartbeater.cc:347] Connected to a master server at 127.22.53.254:36293
I20260926 07:30:29.832304 31276 heartbeater.cc:464] Registering TS with master...
I20260926 07:30:29.832985 31276 heartbeater.cc:511] Master 127.22.53.254:36293 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.22.53.195@KRBTEST.COM; defaulting to no policy
I20260926 07:30:29.834388 30951 ts_manager.cc:194] Registered new tserver with Master: cf15527bf98f4c168972d464e1635d84 (127.22.53.194:42587)
I20260926 07:30:29.835042 30951 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.53.194@KRBTEST.COM'} at 127.22.53.194:58651
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.53.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.53.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.195.keytab.
Entry for principal kudu/127.22.53.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.195.keytab.
I20260926 07:30:29.854104 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.53.195:0
--local_ip_for_outbound_sockets=127.22.53.195
--webserver_interface=127.22.53.195
--webserver_port=0
--tserver_master_addrs=127.22.53.254:36293
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.195.keytab
--principal=kudu/127.22.53.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:45251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:30:29.969733 31281 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:30:29.970117 31281 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:30:29.970207 31281 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:30:29.974736 31281 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:30:29.974864 31281 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:30:29.974964 31281 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.53.195
I20260926 07:30:29.980966 31281 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:45251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.195.keytab
--principal=kudu/127.22.53.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.53.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.53.195
--webserver_port=0
--tserver_master_addrs=127.22.53.254:36293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31281
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.53.195
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:30:29.982259 31281 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:30:29.983453 31281 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:30:29.995193 31289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:29.995208 31287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:29.995265 31281 server_base.cc:1034] running on GCE node
W20260926 07:30:29.995239 31286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:29.996016 31281 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:30:29.996881 31281 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:30:29.998070 31281 hybrid_clock.cc:648] HybridClock initialized: now 1790407829998043 us; error 40 us; skew 500 ppm
Sep 26 07:30:29 dist-test-slave-wdjc krb5kdc[30906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407829, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:30:30.001477 31281 init.cc:377] Logged in from keytab as kudu/127.22.53.195@KRBTEST.COM (short username kudu)
I20260926 07:30:30.002499 31281 webserver.cc:460] Webserver started at http://127.22.53.195:45155/ using document root <none> and password file <none>
I20260926 07:30:30.003151 31281 fs_manager.cc:359] Metadata directory not provided
I20260926 07:30:30.003252 31281 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:30:30.003504 31281 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:30:30.006070 31281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/ts-2/data/instance:
uuid: "2bb3dea3cdd147318c405fc1ebdb9bd6"
format_stamp: "Formatted at 2026-09-26 07:30:29 on dist-test-slave-wdjc"
server_key: "ed83bd40fcf1465b8c5fb4fcbe21cef6"
server_key_iv: "35670c913fb704139ea0c1d92ba3e5c1"
server_key_version: "encryptionkey@0"
I20260926 07:30:30.006738 31281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/ts-2/wal/instance:
uuid: "2bb3dea3cdd147318c405fc1ebdb9bd6"
format_stamp: "Formatted at 2026-09-26 07:30:29 on dist-test-slave-wdjc"
server_key: "ed83bd40fcf1465b8c5fb4fcbe21cef6"
server_key_iv: "35670c913fb704139ea0c1d92ba3e5c1"
server_key_version: "encryptionkey@0"
I20260926 07:30:30.011405 31281 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 07:30:30.014657 31296 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:30.016134 31281 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 07:30:30.016324 31281 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/ts-2/wal
uuid: "2bb3dea3cdd147318c405fc1ebdb9bd6"
format_stamp: "Formatted at 2026-09-26 07:30:29 on dist-test-slave-wdjc"
server_key: "ed83bd40fcf1465b8c5fb4fcbe21cef6"
server_key_iv: "35670c913fb704139ea0c1d92ba3e5c1"
server_key_version: "encryptionkey@0"
I20260926 07:30:30.016479 31281 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:30:30.028429 31281 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:30:30.032209 31281 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:30:30.032449 31281 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:30:30.033129 31281 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:30:30.034063 31281 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:30:30.034139 31281 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:30.034236 31281 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:30:30.034279 31281 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:30.044389 31281 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.195:38907
I20260926 07:30:30.044411 31409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.195:38907 every 8 connection(s)
I20260926 07:30:30.045706 31281 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/ts-2/data/info.pb
I20260926 07:30:30.051767 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 31281
I20260926 07:30:30.051966 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/ts-2/wal/instance
I20260926 07:30:30.052337 22743 external_mini_cluster.cc:1428] Setting key c7a9976ad6db6c71a6759ed6940be4dc
Sep 26 07:30:30 dist-test-slave-wdjc krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407829, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.195@KRBTEST.COM for kudu/127.22.53.254@KRBTEST.COM
I20260926 07:30:30.063194 30990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:30.048511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.195:46385 (local address 127.22.53.254:36293)
0926 07:30:30.048687 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:30.048691 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:30.049946 (+ 1255us) server_negotiation.cc:407] Connection header received
0926 07:30:30.050880 (+ 934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:30.050883 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:30.050935 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:30.051022 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:30.052673 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:30.053410 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:30.054922 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:30.055178 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:30.055304 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:30.058362 (+ 3058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:30.058373 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:30.058376 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:30.058407 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:30.060606 (+ 2199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:30.061312 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:30.061316 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:30.061318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:30.061421 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:30.061888 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:30.061892 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:30.061894 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:30.062123 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:30.062246 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:30.062850 (+ 604us) server_negotiation.cc:299] Negotiation successful
0926 07:30:30.063016 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 07:30:30.064185 31412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:30.048949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:36293 (local address 127.22.53.195:46385)
0926 07:30:30.049763 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:30.049814 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:30.050662 (+ 848us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:30.051263 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:30.051272 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:30.051723 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:30.052395 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:30.052412 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:30.053576 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:30.053583 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:30.054762 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:30.054775 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:30.055375 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:30.055380 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:30.055579 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:30.056327 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:30.056346 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:30.058113 (+ 1767us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:30:30.060806 (+ 2693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:30.060813 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:30.060832 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:30.061172 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:30.061536 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:30.061540 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:30.061542 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:30.061725 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:30.062248 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:30:30.062258 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:30:30.062574 (+ 316us) client_negotiation.cc:769] Sending connection context
0926 07:30:30.062789 (+ 215us) client_negotiation.cc:240] Negotiation successful
0926 07:30:30.063194 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":124,"threads_started":1}
I20260926 07:30:30.065717 31410 heartbeater.cc:347] Connected to a master server at 127.22.53.254:36293
I20260926 07:30:30.066013 31410 heartbeater.cc:464] Registering TS with master...
I20260926 07:30:30.066651 31410 heartbeater.cc:511] Master 127.22.53.254:36293 requested a full tablet report, sending...
I20260926 07:30:30.067936 30951 ts_manager.cc:194] Registered new tserver with Master: 2bb3dea3cdd147318c405fc1ebdb9bd6 (127.22.53.195:38907)
I20260926 07:30:30.068666 30951 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.53.195@KRBTEST.COM'} at 127.22.53.195:46385
I20260926 07:30:30.079629 22743 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.22.53.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.53.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.53.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/krb5kdc/oryx_127.22.53.193.keytab.
Entry for principal oryx/127.22.53.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/krb5kdc/oryx_127.22.53.193.keytab.
WARNING: no policy specified for oryx/127.22.53.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.53.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.53.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/krb5kdc/oryx_127.22.53.194.keytab.
Entry for principal oryx/127.22.53.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/krb5kdc/oryx_127.22.53.194.keytab.
WARNING: no policy specified for oryx/127.22.53.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.53.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.53.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/krb5kdc/oryx_127.22.53.195.keytab.
Entry for principal oryx/127.22.53.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/krb5kdc/oryx_127.22.53.195.keytab.
I20260926 07:30:30.164024 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 31013
I20260926 07:30:30.170199 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 31147
I20260926 07:30:30.175917 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 31281
I20260926 07:30:30.181627 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 30919
I20260926 07:30:30.188889 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.53.254:36293
--webserver_interface=127.22.53.254
--webserver_port=38905
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab
--principal=kudu/127.22.53.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:45251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.53.254:36293
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:30:30.311059 31419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:30:30.311484 31419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:30:30.311574 31419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:30:30.315991 31419 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:30:30.316128 31419 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:30:30.316164 31419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:30:30.316205 31419 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:30:30.316262 31419 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:30:30.322198 31419 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:45251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.53.254:36293
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab
--principal=kudu/127.22.53.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.53.254:36293
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.53.254
--webserver_port=38905
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31419
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:30:30.323467 31419 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:30:30.324631 31419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:30:30.333870 31424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:30.333909 31427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:30.333977 31419 server_base.cc:1034] running on GCE node
W20260926 07:30:30.333871 31425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:30.334743 31419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:30:30.335988 31419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:30:30.337189 31419 hybrid_clock.cc:648] HybridClock initialized: now 1790407830337156 us; error 56 us; skew 500 ppm
Sep 26 07:30:30 dist-test-slave-wdjc krb5kdc[30906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407830, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:30:30.340461 31419 init.cc:377] Logged in from keytab as kudu/127.22.53.254@KRBTEST.COM (short username kudu)
I20260926 07:30:30.341670 31419 webserver.cc:460] Webserver started at http://127.22.53.254:38905/ using document root <none> and password file <none>
I20260926 07:30:30.342320 31419 fs_manager.cc:359] Metadata directory not provided
I20260926 07:30:30.342409 31419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:30:30.347023 31419 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260926 07:30:30.349582 31434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:30.351079 31419 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:30:30.351258 31419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/master-0/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/master-0/wal
uuid: "c96bea3d625746eeab51cbfc0670cf67"
format_stamp: "Formatted at 2026-09-26 07:30:29 on dist-test-slave-wdjc"
server_key: "b262d9ff4f4a14abf80661fe68829b9d"
server_key_iv: "40c90a474a2c0be7301cd5f1c68d84a2"
server_key_version: "encryptionkey@0"
I20260926 07:30:30.351877 31419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:30:30.364872 31419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:30:30.368673 31419 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:30:30.368947 31419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:30:30.376852 31419 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.254:36293
I20260926 07:30:30.376874 31486 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.254:36293 every 8 connection(s)
I20260926 07:30:30.378144 31419 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
I20260926 07:30:30.383118 31487 sys_catalog.cc:263] Verifying existing consensus state
I20260926 07:30:30.385178 31487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c96bea3d625746eeab51cbfc0670cf67: Bootstrap starting.
I20260926 07:30:30.387004 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 31419
I20260926 07:30:30.387516 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.53.193:44951
--local_ip_for_outbound_sockets=127.22.53.193
--tserver_master_addrs=127.22.53.254:36293
--webserver_port=45241
--webserver_interface=127.22.53.193
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.193.keytab
--principal=kudu/127.22.53.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:45251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.22.53.193
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/krb5kdc/oryx_127.22.53.193.keytab with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:30:30.403398 31487 log.cc:824] T 00000000000000000000000000000000 P c96bea3d625746eeab51cbfc0670cf67: Log is configured to *not* fsync() on all Append() calls
I20260926 07:30:30.411156 31487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c96bea3d625746eeab51cbfc0670cf67: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 07:30:30.411635 31487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c96bea3d625746eeab51cbfc0670cf67: Bootstrap complete.
I20260926 07:30:30.416404 31487 raft_consensus.cc:348] T 00000000000000000000000000000000 P c96bea3d625746eeab51cbfc0670cf67 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c96bea3d625746eeab51cbfc0670cf67" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 36293 } }
I20260926 07:30:30.417016 31487 raft_consensus.cc:729] T 00000000000000000000000000000000 P c96bea3d625746eeab51cbfc0670cf67 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c96bea3d625746eeab51cbfc0670cf67, State: Initialized, Role: FOLLOWER
I20260926 07:30:30.417568 31487 consensus_queue.cc:260] T 00000000000000000000000000000000 P c96bea3d625746eeab51cbfc0670cf67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c96bea3d625746eeab51cbfc0670cf67" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 36293 } }
I20260926 07:30:30.417735 31487 raft_consensus.cc:388] T 00000000000000000000000000000000 P c96bea3d625746eeab51cbfc0670cf67 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:30:30.417829 31487 raft_consensus.cc:482] T 00000000000000000000000000000000 P c96bea3d625746eeab51cbfc0670cf67 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:30:30.417955 31487 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c96bea3d625746eeab51cbfc0670cf67 [term 1 FOLLOWER]: Advancing to term 2
I20260926 07:30:30.420252 31487 raft_consensus.cc:504] T 00000000000000000000000000000000 P c96bea3d625746eeab51cbfc0670cf67 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c96bea3d625746eeab51cbfc0670cf67" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 36293 } }
I20260926 07:30:30.420751 31487 leader_election.cc:302] T 00000000000000000000000000000000 P c96bea3d625746eeab51cbfc0670cf67 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c96bea3d625746eeab51cbfc0670cf67; no voters:
I20260926 07:30:30.421157 31487 leader_election.cc:288] T 00000000000000000000000000000000 P c96bea3d625746eeab51cbfc0670cf67 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 07:30:30.421260 31491 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c96bea3d625746eeab51cbfc0670cf67 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 07:30:30.421458 31491 raft_consensus.cc:686] T 00000000000000000000000000000000 P c96bea3d625746eeab51cbfc0670cf67 [term 2 LEADER]: Becoming Leader. State: Replica: c96bea3d625746eeab51cbfc0670cf67, State: Running, Role: LEADER
I20260926 07:30:30.421805 31491 consensus_queue.cc:237] T 00000000000000000000000000000000 P c96bea3d625746eeab51cbfc0670cf67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c96bea3d625746eeab51cbfc0670cf67" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 36293 } }
I20260926 07:30:30.422581 31487 sys_catalog.cc:564] T 00000000000000000000000000000000 P c96bea3d625746eeab51cbfc0670cf67 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:30:30.423309 31492 sys_catalog.cc:455] T 00000000000000000000000000000000 P c96bea3d625746eeab51cbfc0670cf67 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "c96bea3d625746eeab51cbfc0670cf67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c96bea3d625746eeab51cbfc0670cf67" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 36293 } } }
I20260926 07:30:30.423465 31492 sys_catalog.cc:458] T 00000000000000000000000000000000 P c96bea3d625746eeab51cbfc0670cf67 [sys.catalog]: This master's current role is: LEADER
I20260926 07:30:30.423702 31493 sys_catalog.cc:455] T 00000000000000000000000000000000 P c96bea3d625746eeab51cbfc0670cf67 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c96bea3d625746eeab51cbfc0670cf67. Latest consensus state: current_term: 2 leader_uuid: "c96bea3d625746eeab51cbfc0670cf67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c96bea3d625746eeab51cbfc0670cf67" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 36293 } } }
I20260926 07:30:30.423826 31493 sys_catalog.cc:458] T 00000000000000000000000000000000 P c96bea3d625746eeab51cbfc0670cf67 [sys.catalog]: This master's current role is: LEADER
I20260926 07:30:30.424360 31497 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:30:30.428167 31497 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:30:30.428946 31497 catalog_manager.cc:1259] Loaded cluster ID: 5d4899d213984bc2b7a265640ca2e072
I20260926 07:30:30.429020 31497 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:30:30.431547 31497 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:30:30.432305 31497 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c96bea3d625746eeab51cbfc0670cf67: Loaded TSK: 0
I20260926 07:30:30.433679 31497 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 07:30:30.518601 31489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:30:30.518950 31489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:30:30.519164 31489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:30:30.525382 31489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:30:30.525595 31489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:30:30.525722 31489 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.53.193
I20260926 07:30:30.533854 31489 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:45251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/krb5kdc/oryx_127.22.53.193.keytab
--principal=oryx/127.22.53.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.53.193:44951
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.53.193
--webserver_port=45241
--tserver_master_addrs=127.22.53.254:36293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31489
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.53.193
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:30:30.535256 31489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:30:30.536631 31489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:30:30.549901 31513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:30.549903 31516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:30.550045 31514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:30.550350 31489 server_base.cc:1034] running on GCE node
I20260926 07:30:30.550926 31489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:30:30.551860 31489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:30:30.553071 31489 hybrid_clock.cc:648] HybridClock initialized: now 1790407830553050 us; error 56 us; skew 500 ppm
Sep 26 07:30:30 dist-test-slave-wdjc krb5kdc[30906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407830, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.53.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:30:30.557473 31489 init.cc:377] Logged in from keytab as oryx/127.22.53.193@KRBTEST.COM (short username oryx)
I20260926 07:30:30.558881 31489 webserver.cc:460] Webserver started at http://127.22.53.193:45241/ using document root <none> and password file <none>
I20260926 07:30:30.559638 31489 fs_manager.cc:359] Metadata directory not provided
I20260926 07:30:30.559759 31489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:30:30.565552 31489 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 07:30:30.568732 31523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:30.570542 31489 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 07:30:30.570724 31489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/ts-0/wal
uuid: "f01aecc1b8644dd6b7398823c2df278b"
format_stamp: "Formatted at 2026-09-26 07:30:29 on dist-test-slave-wdjc"
server_key: "5d4943cc727c7b35baf469518e7e6a8c"
server_key_iv: "c48e2e45b8037e1f8e681cf8dff6710c"
server_key_version: "encryptionkey@0"
I20260926 07:30:30.571503 31489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:30:30.606770 31489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:30:30.611773 31489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:30:30.612267 31489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:30:30.613307 31489 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:30:30.614328 31489 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:30:30.614384 31489 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:30.614432 31489 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:30:30.614449 31489 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:30.625667 31489 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.193:44951
I20260926 07:30:30.625694 31636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.193:44951 every 8 connection(s)
I20260926 07:30:30.626869 31489 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/ts-0/data/info.pb
I20260926 07:30:30.636422 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 31489
I20260926 07:30:30.636953 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.53.194:42587
--local_ip_for_outbound_sockets=127.22.53.194
--tserver_master_addrs=127.22.53.254:36293
--webserver_port=45857
--webserver_interface=127.22.53.194
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.194.keytab
--principal=kudu/127.22.53.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:45251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.22.53.194
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/krb5kdc/oryx_127.22.53.194.keytab with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 07:30:30 dist-test-slave-wdjc krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.53.193@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:30:30 dist-test-slave-wdjc krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.53.193@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:30:30.641817 31640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:30.629322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:36293 (local address 127.22.53.193:46281)
0926 07:30:30.629870 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:30.629927 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:30.631199 (+ 1272us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:30.632464 (+ 1265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:30.632472 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:30.633021 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:30.633557 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:30.633573 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:30.634990 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:30.634993 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:30.635958 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:30.635970 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:30.636923 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:30.636927 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:30.637131 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:30.637818 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:30.637835 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:30.640688 (+ 2853us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.53.254:36293: Server oryx/127.22.53.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":193,"threads_started":1}
W20260926 07:30:30.641808 31639 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:30:30.629211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.193:46281 (local address 127.22.53.254:36293)
0926 07:30:30.629702 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:30.629719 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:30.630053 (+ 334us) server_negotiation.cc:407] Connection header received
0926 07:30:30.631638 (+ 1585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:30.631650 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:30.631998 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:30.632333 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:30.633795 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:30.634825 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:30.636102 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:30.636711 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:30.636922 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:30.640957 (+ 4035us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.53.193:46281: BlockingRecv error: recv got EOF from 127.22.53.193:46281 (error 108)
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":139,"threads_started":1}
W20260926 07:30:30.642513 31637 heartbeater.cc:644] Failed to heartbeat to 127.22.53.254:36293 (0 consecutive failures): Not authorized: Failed to ping master at 127.22.53.254:36293: Client connection negotiation failed: client connection to 127.22.53.254:36293: Server oryx/127.22.53.254@KRBTEST.COM not found in Kerberos database
Sep 26 07:30:30 dist-test-slave-wdjc krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.53.193@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:30:30 dist-test-slave-wdjc krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.53.193@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:30:30.648700 31640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:30.643017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:36293 (local address 127.22.53.193:35191)
0926 07:30:30.643263 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:30.643278 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:30.643411 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:30.643692 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:30.643695 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:30.643891 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:30.643994 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:30.644000 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:30.644856 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:30.644859 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:30.645431 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:30.645439 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:30.645920 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:30.645924 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:30.646017 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:30.646547 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:30.646560 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:30.648548 (+ 1988us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.53.254:36293: Server oryx/127.22.53.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178}
W20260926 07:30:30.648975 31639 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:30:30.643135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.193:35191 (local address 127.22.53.254:36293)
0926 07:30:30.643250 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:30.643254 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:30.643352 (+ 98us) server_negotiation.cc:407] Connection header received
0926 07:30:30.643518 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:30.643521 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:30.643574 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:30.643675 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:30.644114 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:30.644737 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:30.645563 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:30.645793 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:30.645905 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:30.648854 (+ 2949us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.53.193:35191: BlockingRecv error: recv got EOF from 127.22.53.193:35191 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Sep 26 07:30:30 dist-test-slave-wdjc krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.53.193@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:30:30 dist-test-slave-wdjc krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.53.193@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:30:30.655690 31640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:30.649433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:36293 (local address 127.22.53.193:40529)
0926 07:30:30.649583 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:30.649598 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:30.649692 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:30.650112 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:30.650115 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:30.650262 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:30.650346 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:30.650351 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:30.651472 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:30.651474 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:30.652119 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:30.652130 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:30.652769 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:30.652772 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:30.652841 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:30.653390 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:30.653402 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:30.655570 (+ 2168us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.53.254:36293: Server oryx/127.22.53.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":65}
W20260926 07:30:30.655898 31639 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:30:30.649590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.193:40529 (local address 127.22.53.254:36293)
0926 07:30:30.649840 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:30.649845 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:30.649861 (+ 16us) server_negotiation.cc:407] Connection header received
0926 07:30:30.649905 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:30.649908 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:30.649953 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:30.650046 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:30.650503 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:30.651350 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:30.652246 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:30.652634 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:30.652730 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:30.655773 (+ 3043us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.53.193:40529: BlockingRecv error: recv got EOF from 127.22.53.193:40529 (error 108)
Metrics: {"server-negotiator.queue_time_us":155}
W20260926 07:30:30.656047 31637 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 07:30:30.759878 31641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:30:30.760185 31641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:30:30.760239 31641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:30:30.764787 31641 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:30:30.764868 31641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:30:30.764998 31641 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.53.194
I20260926 07:30:30.770783 31641 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:45251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/krb5kdc/oryx_127.22.53.194.keytab
--principal=oryx/127.22.53.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.53.194:42587
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.53.194
--webserver_port=45857
--tserver_master_addrs=127.22.53.254:36293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31641
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.53.194
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:30:30.772040 31641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:30:30.773172 31641 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:30:30.783931 31649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:30.783931 31646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:30.783959 31647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:30.783993 31641 server_base.cc:1034] running on GCE node
I20260926 07:30:30.784700 31641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:30:30.785470 31641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:30:30.786715 31641 hybrid_clock.cc:648] HybridClock initialized: now 1790407830786699 us; error 35 us; skew 500 ppm
Sep 26 07:30:30 dist-test-slave-wdjc krb5kdc[30906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407830, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.53.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:30:30.789954 31641 init.cc:377] Logged in from keytab as oryx/127.22.53.194@KRBTEST.COM (short username oryx)
I20260926 07:30:30.790995 31641 webserver.cc:460] Webserver started at http://127.22.53.194:45857/ using document root <none> and password file <none>
I20260926 07:30:30.791615 31641 fs_manager.cc:359] Metadata directory not provided
I20260926 07:30:30.791743 31641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:30:30.796139 31641 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 07:30:30.798682 31656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:30.800083 31641 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 07:30:30.800269 31641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/ts-1/wal
uuid: "cf15527bf98f4c168972d464e1635d84"
format_stamp: "Formatted at 2026-09-26 07:30:29 on dist-test-slave-wdjc"
server_key: "d58727c1760d61980c4035fad02d9d1a"
server_key_iv: "11ec095ce093578a1c9a2193ce90f0c1"
server_key_version: "encryptionkey@0"
I20260926 07:30:30.800999 31641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:30:30.814697 31641 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:30:30.818639 31641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:30:30.818890 31641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:30:30.819635 31641 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:30:30.820501 31641 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:30:30.820574 31641 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:30.820618 31641 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:30:30.820652 31641 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:30.830494 31641 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.194:42587
I20260926 07:30:30.830554 31769 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.194:42587 every 8 connection(s)
I20260926 07:30:30.831678 31641 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/ts-1/data/info.pb
I20260926 07:30:30.834882 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 31641
I20260926 07:30:30.835388 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.53.195:38907
--local_ip_for_outbound_sockets=127.22.53.195
--tserver_master_addrs=127.22.53.254:36293
--webserver_port=45155
--webserver_interface=127.22.53.195
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.195.keytab
--principal=kudu/127.22.53.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:45251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.22.53.195
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/krb5kdc/oryx_127.22.53.195.keytab with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 07:30:30 dist-test-slave-wdjc krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.53.194@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:30:30 dist-test-slave-wdjc krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.53.194@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM, Server not found in Kerberos database
W20260926 07:30:30.844692 31639 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:30:30.833769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.194:54117 (local address 127.22.53.254:36293)
0926 07:30:30.833943 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:30.833948 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:30.834975 (+ 1027us) server_negotiation.cc:407] Connection header received
0926 07:30:30.836579 (+ 1604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:30.836583 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:30.836664 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:30.836774 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:30.838053 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:30.838589 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:30.839832 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:30.840075 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:30.840197 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:30.844526 (+ 4329us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.53.194:54117: BlockingRecv error: recv got EOF from 127.22.53.194:54117 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 07:30:30.845486 31772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:30.834272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:36293 (local address 127.22.53.194:54117)
0926 07:30:30.834790 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:30.834835 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:30.836304 (+ 1469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:30.836977 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:30.836985 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:30.837381 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:30.837889 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:30.837903 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:30.838747 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:30.838750 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:30.839686 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:30.839698 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:30.840254 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:30.840258 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:30.840451 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:30.841342 (+ 891us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:30.841360 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:30.844362 (+ 3002us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.53.254:36293: Server oryx/127.22.53.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":120,"threads_started":1}
W20260926 07:30:30.846016 31770 heartbeater.cc:644] Failed to heartbeat to 127.22.53.254:36293 (0 consecutive failures): Not authorized: Failed to ping master at 127.22.53.254:36293: Client connection negotiation failed: client connection to 127.22.53.254:36293: Server oryx/127.22.53.254@KRBTEST.COM not found in Kerberos database
Sep 26 07:30:30 dist-test-slave-wdjc krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.53.194@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:30:30 dist-test-slave-wdjc krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.53.194@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:30:30.853297 31772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:30.846691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:36293 (local address 127.22.53.194:52869)
0926 07:30:30.846888 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:30.846904 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:30.846997 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:30.847347 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:30.847350 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:30.847548 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:30.847677 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:30.847683 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:30.848671 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:30.848674 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:30.849607 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:30.849618 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:30.850111 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:30.850114 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:30.850212 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:30.850792 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:30.850806 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:30.853110 (+ 2304us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.53.254:36293: Server oryx/127.22.53.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123}
W20260926 07:30:30.853426 31639 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:30:30.846811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.194:52869 (local address 127.22.53.254:36293)
0926 07:30:30.846939 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:30.846942 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:30.846957 (+ 15us) server_negotiation.cc:407] Connection header received
0926 07:30:30.847140 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:30.847143 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:30.847195 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:30.847299 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:30.847821 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:30.848480 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:30.849753 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:30.849981 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:30.850084 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:30.853284 (+ 3200us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.53.194:52869: BlockingRecv error: recv got EOF from 127.22.53.194:52869 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 26 07:30:30 dist-test-slave-wdjc krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.53.194@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:30:30 dist-test-slave-wdjc krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.53.194@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:30:30.860471 31772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:30.854110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:36293 (local address 127.22.53.194:59103)
0926 07:30:30.854325 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:30.854339 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:30.854418 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:30.854748 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:30.854751 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:30.855020 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:30.855114 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:30.855120 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:30.856277 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:30.856280 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:30.857013 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:30.857026 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:30.857711 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:30.857713 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:30.857783 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:30.858399 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:30.858412 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:30.860349 (+ 1937us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.53.254:36293: Server oryx/127.22.53.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":72}
W20260926 07:30:30.860785 31639 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:30:30.854219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.194:59103 (local address 127.22.53.254:36293)
0926 07:30:30.854366 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:30.854371 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:30.854389 (+ 18us) server_negotiation.cc:407] Connection header received
0926 07:30:30.854518 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:30.854521 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:30.854572 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:30.854659 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:30.855245 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:30.856102 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:30.857182 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:30.857565 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:30.857678 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:30.860645 (+ 2967us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.53.194:59103: BlockingRecv error: recv got EOF from 127.22.53.194:59103 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260926 07:30:30.860808 31770 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 07:30:30.959125 31773 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:30:30.959453 31773 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:30:30.959501 31773 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:30:30.963999 31773 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:30:30.964092 31773 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:30:30.964156 31773 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.53.195
I20260926 07:30:30.970013 31773 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:45251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/krb5kdc/oryx_127.22.53.195.keytab
--principal=oryx/127.22.53.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.53.195:38907
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.53.195
--webserver_port=45155
--tserver_master_addrs=127.22.53.254:36293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31773
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.53.195
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:30:30.971243 31773 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:30:30.972352 31773 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:30:30.983096 31779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:30.983065 31778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:30.983222 31781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:30.983330 31773 server_base.cc:1034] running on GCE node
I20260926 07:30:30.983826 31773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:30:30.984627 31773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:30:30.985800 31773 hybrid_clock.cc:648] HybridClock initialized: now 1790407830985787 us; error 29 us; skew 500 ppm
Sep 26 07:30:30 dist-test-slave-wdjc krb5kdc[30906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407830, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.53.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:30:30.988955 31773 init.cc:377] Logged in from keytab as oryx/127.22.53.195@KRBTEST.COM (short username oryx)
I20260926 07:30:30.990161 31773 webserver.cc:460] Webserver started at http://127.22.53.195:45155/ using document root <none> and password file <none>
I20260926 07:30:30.990826 31773 fs_manager.cc:359] Metadata directory not provided
I20260926 07:30:30.990962 31773 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:30:30.995620 31773 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 07:30:30.998099 31788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:30.999554 31773 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 07:30:30.999739 31773 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/ts-2/wal
uuid: "2bb3dea3cdd147318c405fc1ebdb9bd6"
format_stamp: "Formatted at 2026-09-26 07:30:29 on dist-test-slave-wdjc"
server_key: "ed83bd40fcf1465b8c5fb4fcbe21cef6"
server_key_iv: "35670c913fb704139ea0c1d92ba3e5c1"
server_key_version: "encryptionkey@0"
I20260926 07:30:31.000358 31773 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:30:31.023496 31773 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:30:31.026930 31773 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:30:31.027155 31773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:30:31.027796 31773 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:30:31.028698 31773 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:30:31.028746 31773 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:31.028820 31773 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:30:31.028856 31773 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:31.038352 31773 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.195:38907
I20260926 07:30:31.038368 31901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.195:38907 every 8 connection(s)
I20260926 07:30:31.039523 31773 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407783529717-22743-0/minicluster-data/ts-2/data/info.pb
I20260926 07:30:31.042984 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 31773
Sep 26 07:30:31 dist-test-slave-wdjc krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.53.195@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:30:31 dist-test-slave-wdjc krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.53.195@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM, Server not found in Kerberos database
W20260926 07:30:31.052618 31639 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:30:31.041676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.195:59097 (local address 127.22.53.254:36293)
0926 07:30:31.041867 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:31.041871 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:31.042738 (+ 867us) server_negotiation.cc:407] Connection header received
0926 07:30:31.044358 (+ 1620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:31.044361 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:31.044406 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:31.044472 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:31.045855 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:31.046529 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:31.048052 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:31.048271 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:31.048381 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:31.052430 (+ 4049us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.53.195:59097: BlockingRecv error: recv got EOF from 127.22.53.195:59097 (error 108)
Metrics: {"server-negotiator.queue_time_us":89,"spinlock_wait_cycles":1408}
I20260926 07:30:31.053357 31904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:31.042081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:36293 (local address 127.22.53.195:59097)
0926 07:30:31.042555 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:31.042598 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:31.044111 (+ 1513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:31.044721 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:31.044728 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:31.045078 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:31.045690 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:31.045710 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:31.046699 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:31.046702 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:31.047922 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:31.047932 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:31.048459 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:31.048466 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:31.048708 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:31.049292 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:31.049309 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:31.052248 (+ 2939us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.53.254:36293: Server oryx/127.22.53.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":118,"threads_started":1}
W20260926 07:30:31.053902 31902 heartbeater.cc:644] Failed to heartbeat to 127.22.53.254:36293 (0 consecutive failures): Not authorized: Failed to ping master at 127.22.53.254:36293: Client connection negotiation failed: client connection to 127.22.53.254:36293: Server oryx/127.22.53.254@KRBTEST.COM not found in Kerberos database
I20260926 07:30:31.054490 31906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:31.044052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50300 (local address 127.22.53.254:36293)
0926 07:30:31.044384 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:31.044388 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:31.044402 (+ 14us) server_negotiation.cc:407] Connection header received
0926 07:30:31.044442 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:31.044447 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:31.044494 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:31.044604 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:31.045336 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:31.046067 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:31.047127 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:31.047470 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:31.047587 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:31.049501 (+ 1914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:31.049510 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:31.049536 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:31.049569 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:31.051529 (+ 1960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:31.052127 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:31.052133 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:31.052138 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:31.052339 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:31.052689 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:31.052692 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:31.052694 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:31.053158 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:31.053357 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:31.054033 (+ 676us) server_negotiation.cc:299] Negotiation successful
0926 07:30:31.054294 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":102,"threads_started":1}
Sep 26 07:30:31 dist-test-slave-wdjc krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.53.195@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:30:31 dist-test-slave-wdjc krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.53.195@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:30:31.060922 31904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:31.054391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:36293 (local address 127.22.53.195:54421)
0926 07:30:31.054533 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:31.054549 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:31.055459 (+ 910us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:31.055743 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:31.055746 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:31.055929 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:31.056031 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:31.056037 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:31.057032 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:31.057034 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:31.057818 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:31.057830 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:31.058243 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:31.058246 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:31.058317 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:31.058787 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:31.058799 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:31.060752 (+ 1953us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.53.254:36293: Server oryx/127.22.53.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":51}
W20260926 07:30:31.061028 31639 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:30:31.054511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.195:54421 (local address 127.22.53.254:36293)
0926 07:30:31.054660 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:31.054665 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:31.055398 (+ 733us) server_negotiation.cc:407] Connection header received
0926 07:30:31.055559 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:31.055580 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:31.055632 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:31.055725 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:31.056149 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:31.056917 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:31.057930 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:31.058141 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:31.058224 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:31.060912 (+ 2688us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.53.195:54421: BlockingRecv error: recv got EOF from 127.22.53.195:54421 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 26 07:30:31 dist-test-slave-wdjc krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.53.195@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:30:31 dist-test-slave-wdjc krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.53.195@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:30:31.067749 31904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:31.061816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:36293 (local address 127.22.53.195:38243)
0926 07:30:31.061944 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:31.061960 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:31.062044 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:31.062376 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:31.062428 (+ 52us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:31.062674 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:31.062784 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:31.062791 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:31.063784 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:31.063787 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:31.064345 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:31.064354 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:31.065029 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:31.065031 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:31.065095 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:31.065756 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:31.065779 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:31.067590 (+ 1811us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.53.254:36293: Server oryx/127.22.53.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":55}
W20260926 07:30:31.067986 31639 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:30:31.061880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.195:38243 (local address 127.22.53.254:36293)
0926 07:30:31.062041 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:31.062044 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:31.062057 (+ 13us) server_negotiation.cc:407] Connection header received
0926 07:30:31.062172 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:31.062174 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:31.062226 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:31.062324 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:31.062910 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:31.063637 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:31.064509 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:31.064906 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:31.064992 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:31.067839 (+ 2847us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.53.195:38243: BlockingRecv error: recv got EOF from 127.22.53.195:38243 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
W20260926 07:30:31.068665 31902 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 26 07:30:31 dist-test-slave-wdjc krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.53.193@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:30:31 dist-test-slave-wdjc krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.53.193@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:30:31.662945 31907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:31.656791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:36293 (local address 127.22.53.193:41693)
0926 07:30:31.657173 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:31.657189 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:31.657292 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:31.657582 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:31.657584 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:31.657794 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:31.657917 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:31.657923 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:31.658739 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:31.658742 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:31.659345 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:31.659354 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:31.659850 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:31.659853 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:31.659933 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:31.660519 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:31.660562 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:31.662771 (+ 2209us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.53.254:36293: Server oryx/127.22.53.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":113,"threads_started":1}
W20260926 07:30:31.663093 31908 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:30:31.656945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.193:41693 (local address 127.22.53.254:36293)
0926 07:30:31.657219 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:31.657222 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:31.657243 (+ 21us) server_negotiation.cc:407] Connection header received
0926 07:30:31.657393 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:31.657396 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:31.657441 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:31.657521 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:31.658062 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:31.658613 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:31.659467 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:31.659698 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:31.659786 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:31.662944 (+ 3158us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.53.193:41693: BlockingRecv error: recv got EOF from 127.22.53.193:41693 (error 108)
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":73,"threads_started":1}
Sep 26 07:30:31 dist-test-slave-wdjc krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.53.194@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:30:31 dist-test-slave-wdjc krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.53.194@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:30:31.867693 31909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:31.861484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:36293 (local address 127.22.53.194:46353)
0926 07:30:31.861760 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:31.861773 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:31.861874 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:31.862116 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:31.862118 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:31.862304 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:31.862421 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:31.862427 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:31.863285 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:31.863288 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:31.863888 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:31.863898 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:31.864425 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:31.864428 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:31.864497 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:31.865260 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:31.865272 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:31.867552 (+ 2280us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.53.254:36293: Server oryx/127.22.53.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":100,"threads_started":1}
W20260926 07:30:31.867892 31908 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:30:31.861600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.194:46353 (local address 127.22.53.254:36293)
0926 07:30:31.861751 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:31.861755 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:31.861843 (+ 88us) server_negotiation.cc:407] Connection header received
0926 07:30:31.861954 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:31.861956 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:31.862000 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:31.862072 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:31.862580 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:31.863147 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:31.864042 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:31.864296 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:31.864397 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:31.867739 (+ 3342us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.53.194:46353: BlockingRecv error: recv got EOF from 127.22.53.194:46353 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 26 07:30:32 dist-test-slave-wdjc krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.53.195@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:30:32 dist-test-slave-wdjc krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.53.195@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:30:32.075950 31910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:32.069598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:36293 (local address 127.22.53.195:38893)
0926 07:30:32.070134 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:32.070150 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:32.070304 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:32.070692 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:32.070695 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:32.070907 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:32.071037 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:32.071043 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:32.071812 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:32.071815 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:32.072409 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:32.072419 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:32.073006 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:32.073010 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:32.073091 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:32.073743 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:32.073755 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:32.075817 (+ 2062us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.53.254:36293: Server oryx/127.22.53.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":122,"threads_started":1}
W20260926 07:30:32.076253 31908 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:30:32.069676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.195:38893 (local address 127.22.53.254:36293)
0926 07:30:32.069840 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:32.069844 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:32.070262 (+ 418us) server_negotiation.cc:407] Connection header received
0926 07:30:32.070485 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:32.070488 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:32.070565 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:32.070649 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:32.071155 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:32.071688 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:32.072547 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:32.072814 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:32.072898 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:32.076122 (+ 3224us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.53.195:38893: BlockingRecv error: recv got EOF from 127.22.53.195:38893 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 26 07:30:32 dist-test-slave-wdjc krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.53.193@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:30:32 dist-test-slave-wdjc krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.53.193@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:30:32.670035 31911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:32.664009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:36293 (local address 127.22.53.193:48499)
0926 07:30:32.664311 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:32.664326 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:32.664424 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:32.664721 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:32.664723 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:32.664918 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:32.665019 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:32.665025 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:32.665852 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:32.665855 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:32.666445 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:32.666455 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:32.666954 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:32.666957 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:32.667027 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:32.667580 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:32.667592 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:32.669845 (+ 2253us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.53.254:36293: Server oryx/127.22.53.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":111,"threads_started":1}
W20260926 07:30:32.670284 31912 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:30:32.664117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.193:48499 (local address 127.22.53.254:36293)
0926 07:30:32.664442 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:32.664446 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:32.664462 (+ 16us) server_negotiation.cc:407] Connection header received
0926 07:30:32.664517 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:32.664519 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:32.664586 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:32.664681 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:32.665143 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:32.665711 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:32.666569 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:32.666815 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:32.666900 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:32.670113 (+ 3213us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.53.193:48499: BlockingRecv error: recv got EOF from 127.22.53.193:48499 (error 108)
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":143,"threads_started":1}
Sep 26 07:30:32 dist-test-slave-wdjc krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.53.194@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:30:32 dist-test-slave-wdjc krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.53.194@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:30:32.878386 31913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:32.868656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:36293 (local address 127.22.53.194:60515)
0926 07:30:32.868984 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:32.868997 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:32.869144 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:32.869439 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:32.869441 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:32.869664 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:32.869762 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:32.869768 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:32.870590 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:32.870595 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:32.871176 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:32.871185 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:32.871674 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:32.871678 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:32.871767 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:32.875982 (+ 4215us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:32.875996 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:32.878217 (+ 2221us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.53.254:36293: Server oryx/127.22.53.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":91,"threads_started":1}
W20260926 07:30:32.878588 31912 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:30:32.868722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.194:60515 (local address 127.22.53.254:36293)
0926 07:30:32.868923 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:32.868928 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:32.869080 (+ 152us) server_negotiation.cc:407] Connection header received
0926 07:30:32.869266 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:32.869269 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:32.869321 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:32.869412 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:32.869906 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:32.870467 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:32.871315 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:32.871544 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:32.871663 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:32.878470 (+ 6807us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.53.194:60515: BlockingRecv error: recv got EOF from 127.22.53.194:60515 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
Sep 26 07:30:33 dist-test-slave-wdjc krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.53.195@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:30:33 dist-test-slave-wdjc krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.53.195@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:30:33.083400 31914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:33.076999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:36293 (local address 127.22.53.195:44971)
0926 07:30:33.077465 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:33.077481 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:33.077641 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:33.077934 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:33.077937 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:33.078167 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:33.078268 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:33.078274 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:33.079279 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:33.079285 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:33.080055 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:33.080065 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:33.080606 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:33.080613 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:33.080675 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:33.081240 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:33.081252 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:33.083254 (+ 2002us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.53.254:36293: Server oryx/127.22.53.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":251,"threads_started":1}
W20260926 07:30:33.083619 31912 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:30:33.077299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.195:44971 (local address 127.22.53.254:36293)
0926 07:30:33.077465 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:33.077471 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:33.077571 (+ 100us) server_negotiation.cc:407] Connection header received
0926 07:30:33.077763 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:33.077766 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:33.077823 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:33.077912 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:33.078476 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:33.079142 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:33.080193 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:33.080449 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:33.080570 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:33.083460 (+ 2890us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.53.195:44971: BlockingRecv error: recv got EOF from 127.22.53.195:44971 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 26 07:30:33 dist-test-slave-wdjc krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.53.193@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:30:33 dist-test-slave-wdjc krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.53.193@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:30:33.677593 31915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:33.671200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:36293 (local address 127.22.53.193:37805)
0926 07:30:33.671532 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:33.671550 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:33.671659 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:33.672046 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:33.672049 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:33.672266 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:33.672364 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:33.672370 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:33.673266 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:33.673271 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:33.673873 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:33.673883 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:33.674397 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:33.674401 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:33.674493 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:33.675098 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:33.675109 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:33.677387 (+ 2278us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.53.254:36293: Server oryx/127.22.53.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":130,"threads_started":1}
W20260926 07:30:33.677776 31916 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:30:33.671362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.193:37805 (local address 127.22.53.254:36293)
0926 07:30:33.671752 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:33.671759 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:33.671779 (+ 20us) server_negotiation.cc:407] Connection header received
0926 07:30:33.671838 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:33.671840 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:33.671884 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:33.672014 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:33.672502 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:33.673137 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:33.674025 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:33.674281 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:33.674391 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:33.677599 (+ 3208us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.53.193:37805: BlockingRecv error: recv got EOF from 127.22.53.193:37805 (error 108)
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":138,"threads_started":1}
Sep 26 07:30:33 dist-test-slave-wdjc krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.53.194@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:30:33 dist-test-slave-wdjc krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.53.194@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:30:33.886116 31917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:33.879350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:36293 (local address 127.22.53.194:60567)
0926 07:30:33.879641 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:33.879657 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:33.879768 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:33.880062 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:33.880064 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:33.880253 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:33.880407 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:33.880414 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:33.881288 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:33.881291 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:33.881891 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:33.881902 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:33.882543 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:33.882549 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:33.882670 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:33.883548 (+ 878us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:33.883569 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:33.885984 (+ 2415us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.53.254:36293: Server oryx/127.22.53.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":108,"threads_started":1}
W20260926 07:30:33.886318 31916 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:30:33.879534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.194:60567 (local address 127.22.53.254:36293)
0926 07:30:33.879694 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:33.879698 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:33.879719 (+ 21us) server_negotiation.cc:407] Connection header received
0926 07:30:33.879869 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:33.879871 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:33.879918 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:33.880020 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:33.880573 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:33.881140 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:33.882028 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:33.882282 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:33.882373 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:33.886151 (+ 3778us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.53.194:60567: BlockingRecv error: recv got EOF from 127.22.53.194:60567 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 26 07:30:34 dist-test-slave-wdjc krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.53.195@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:30:34 dist-test-slave-wdjc krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.53.195@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:30:34.090301 31918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:34.084417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:36293 (local address 127.22.53.195:55547)
0926 07:30:34.084724 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:34.084740 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:34.084853 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:34.085112 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:34.085114 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:34.085297 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:34.085391 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:34.085396 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:34.086215 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:34.086218 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:34.086830 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:34.086841 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:34.087307 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:34.087310 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:34.087375 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:34.087920 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:34.087933 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:34.090159 (+ 2226us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.53.254:36293: Server oryx/127.22.53.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":134,"threads_started":1}
W20260926 07:30:34.090528 31916 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:30:34.084612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.195:55547 (local address 127.22.53.254:36293)
0926 07:30:34.084765 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:34.084768 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:34.084801 (+ 33us) server_negotiation.cc:407] Connection header received
0926 07:30:34.084939 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:34.084941 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:34.084992 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:34.085073 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:34.085498 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:34.086084 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:34.086951 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:34.087187 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:34.087278 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:34.090366 (+ 3088us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.53.195:55547: BlockingRecv error: recv got EOF from 127.22.53.195:55547 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 26 07:30:34 dist-test-slave-wdjc krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.53.193@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:30:34 dist-test-slave-wdjc krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.53.193@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:30:34.684952 31919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:34.678558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:36293 (local address 127.22.53.193:46695)
0926 07:30:34.678903 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:34.678924 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:34.679061 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:34.679389 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:34.679392 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:34.679594 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:34.679708 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:34.679714 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:34.680663 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:34.680666 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:34.681239 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:34.681248 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:34.681784 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:34.681787 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:34.681861 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:34.682687 (+ 826us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:34.682700 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:34.684822 (+ 2122us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.53.254:36293: Server oryx/127.22.53.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":73,"threads_started":1}
W20260926 07:30:34.685262 31920 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:30:34.678753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.193:46695 (local address 127.22.53.254:36293)
0926 07:30:34.679024 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:34.679028 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:34.679043 (+ 15us) server_negotiation.cc:407] Connection header received
0926 07:30:34.679154 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:34.679156 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:34.679202 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:34.679308 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:34.679917 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:34.680509 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:34.681374 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:34.681620 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:34.681728 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:34.685034 (+ 3306us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.53.193:46695: BlockingRecv error: recv got EOF from 127.22.53.193:46695 (error 108)
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":87,"threads_started":1}
Sep 26 07:30:34 dist-test-slave-wdjc krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.53.194@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:30:34 dist-test-slave-wdjc krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.53.194@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:30:34.893352 31921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:34.887158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:36293 (local address 127.22.53.194:60491)
0926 07:30:34.887520 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:34.887536 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:34.887706 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:34.888015 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:34.888018 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:34.888245 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:34.888347 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:34.888353 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:34.889230 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:34.889235 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:34.889856 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:34.889866 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:34.890360 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:34.890364 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:34.890459 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:34.891101 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:34.891113 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:34.893160 (+ 2047us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.53.254:36293: Server oryx/127.22.53.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":93,"threads_started":1}
W20260926 07:30:34.893469 31920 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:30:34.887323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.194:60491 (local address 127.22.53.254:36293)
0926 07:30:34.887507 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:34.887512 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:34.887639 (+ 127us) server_negotiation.cc:407] Connection header received
0926 07:30:34.887845 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:34.887847 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:34.887893 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:34.887989 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:34.888504 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:34.889110 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:34.889992 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:34.890229 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:34.890352 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:34.893310 (+ 2958us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.53.194:60491: BlockingRecv error: recv got EOF from 127.22.53.194:60491 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 26 07:30:35 dist-test-slave-wdjc krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.53.195@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:30:35 dist-test-slave-wdjc krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.53.195@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:30:35.099301 31922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:35.091881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:36293 (local address 127.22.53.195:45831)
0926 07:30:35.092218 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:35.092234 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:35.092353 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:35.092803 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:35.092809 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:35.093078 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:35.093241 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:35.093249 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:35.094312 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:35.094317 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:35.095071 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:35.095084 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:35.095698 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:35.095701 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:35.095784 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:35.096424 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:35.096441 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:35.099131 (+ 2690us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.53.254:36293: Server oryx/127.22.53.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":108,"threads_started":1}
W20260926 07:30:35.099514 31920 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:30:35.092006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.195:45831 (local address 127.22.53.254:36293)
0926 07:30:35.092237 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:35.092243 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:35.092313 (+ 70us) server_negotiation.cc:407] Connection header received
0926 07:30:35.092459 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:35.092461 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:35.092519 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:35.092651 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:35.093487 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:35.094083 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:35.095279 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:35.095544 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:35.095643 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:35.099347 (+ 3704us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.53.195:45831: BlockingRecv error: recv got EOF from 127.22.53.195:45831 (error 108)
Metrics: {"server-negotiator.queue_time_us":111}
Sep 26 07:30:35 dist-test-slave-wdjc krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.53.193@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:30:35 dist-test-slave-wdjc krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.53.193@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:30:35.692162 31923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:35.685951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:36293 (local address 127.22.53.193:35927)
0926 07:30:35.686362 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:35.686377 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:35.686480 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:35.686804 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:35.686807 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:35.687000 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:35.687103 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:35.687109 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:35.687947 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:35.687950 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:35.688778 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:35.688790 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:35.689365 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:35.689369 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:35.689436 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:35.690081 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:35.690102 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:35.692030 (+ 1928us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.53.254:36293: Server oryx/127.22.53.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":115,"threads_started":1}
W20260926 07:30:35.692390 31924 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:30:35.686139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.193:35927 (local address 127.22.53.254:36293)
0926 07:30:35.686434 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:35.686438 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:35.686454 (+ 16us) server_negotiation.cc:407] Connection header received
0926 07:30:35.686584 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:35.686586 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:35.686633 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:35.686734 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:35.687240 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:35.687816 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:35.688888 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:35.689224 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:35.689345 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:35.692209 (+ 2864us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.53.193:35927: BlockingRecv error: recv got EOF from 127.22.53.193:35927 (error 108)
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":80,"threads_started":1}
Sep 26 07:30:35 dist-test-slave-wdjc krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.53.194@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:30:35 dist-test-slave-wdjc krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.53.194@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:30:35.901119 31925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:35.894641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:36293 (local address 127.22.53.194:46165)
0926 07:30:35.895117 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:35.895135 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:35.895272 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:35.895583 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:35.895587 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:35.895866 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:35.895973 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:35.895979 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:35.897193 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:35.897196 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:35.897813 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:35.897823 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:35.898347 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:35.898350 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:35.898415 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:35.898969 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:35.898981 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:35.900989 (+ 2008us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.53.254:36293: Server oryx/127.22.53.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":143,"threads_started":1}
W20260926 07:30:35.901420 31924 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:30:35.894872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.194:46165 (local address 127.22.53.254:36293)
0926 07:30:35.895012 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:35.895016 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:35.895251 (+ 235us) server_negotiation.cc:407] Connection header received
0926 07:30:35.895384 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:35.895389 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:35.895445 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:35.895557 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:35.896129 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:35.897056 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:35.897959 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:35.898224 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:35.898326 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:35.901289 (+ 2963us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.53.194:46165: BlockingRecv error: recv got EOF from 127.22.53.194:46165 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 26 07:30:36 dist-test-slave-wdjc krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.53.195@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:30:36 dist-test-slave-wdjc krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.53.195@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:30:36.108461 31926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:36.100345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:36293 (local address 127.22.53.195:45735)
0926 07:30:36.100653 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:36.100673 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:36.100800 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:36.101364 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:36.101368 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:36.101643 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:36.101753 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:36.101760 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:36.102656 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:36.102660 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:36.103541 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:36.103554 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:36.104338 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:36.104342 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:36.104457 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:36.105194 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:36.105211 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:36.108256 (+ 3045us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.53.254:36293: Server oryx/127.22.53.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":85,"threads_started":1}
W20260926 07:30:36.108701 31924 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:30:36.100868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.195:45735 (local address 127.22.53.254:36293)
0926 07:30:36.101046 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:36.101052 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:36.101072 (+ 20us) server_negotiation.cc:407] Connection header received
0926 07:30:36.101138 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:36.101141 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:36.101228 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:36.101318 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:36.101895 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:36.102511 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:36.103736 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:36.104204 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:36.104326 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:36.108534 (+ 4208us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.53.195:45735: BlockingRecv error: recv got EOF from 127.22.53.195:45735 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 26 07:30:36 dist-test-slave-wdjc krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.53.193@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:30:36 dist-test-slave-wdjc krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.53.193@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:30:36.699723 31927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:36.693112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:36293 (local address 127.22.53.193:49511)
0926 07:30:36.693457 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:36.693470 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:36.693590 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:36.693823 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:36.693825 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:36.694014 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:36.694165 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:36.694171 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:36.694992 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:36.694996 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:36.695640 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:36.695652 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:36.696254 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:36.696257 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:36.696329 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:36.697312 (+ 983us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:36.697325 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:36.699552 (+ 2227us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.53.254:36293: Server oryx/127.22.53.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":124,"threads_started":1}
W20260926 07:30:36.699954 31928 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:30:36.693311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.193:49511 (local address 127.22.53.254:36293)
0926 07:30:36.693588 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:36.693592 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:36.693606 (+ 14us) server_negotiation.cc:407] Connection header received
0926 07:30:36.693658 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:36.693660 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:36.693703 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:36.693786 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:36.694284 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:36.694835 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:36.695798 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:36.696103 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:36.696202 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:36.699785 (+ 3583us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.53.193:49511: BlockingRecv error: recv got EOF from 127.22.53.193:49511 (error 108)
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":104,"threads_started":1}
Sep 26 07:30:36 dist-test-slave-wdjc krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.53.194@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:30:36 dist-test-slave-wdjc krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.53.194@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:30:36.908213 31929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:36.902075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:36293 (local address 127.22.53.194:51367)
0926 07:30:36.902348 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:36.902361 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:36.902464 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:36.902746 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:36.902748 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:36.902947 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:36.903097 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:36.903103 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:36.904152 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:36.904156 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:36.904861 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:36.904873 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:36.905375 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:36.905378 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:36.905443 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:36.906084 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:36.906096 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:36.908083 (+ 1987us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.53.254:36293: Server oryx/127.22.53.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":96,"threads_started":1}
W20260926 07:30:36.908509 31928 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:30:36.902168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.194:51367 (local address 127.22.53.254:36293)
0926 07:30:36.902332 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:36.902336 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:36.902436 (+ 100us) server_negotiation.cc:407] Connection header received
0926 07:30:36.902550 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:36.902552 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:36.902599 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:36.902674 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:36.903273 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:36.903912 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:36.904990 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:36.905226 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:36.905363 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:36.908371 (+ 3008us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.53.194:51367: BlockingRecv error: recv got EOF from 127.22.53.194:51367 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 26 07:30:37 dist-test-slave-wdjc krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.53.195@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:30:37 dist-test-slave-wdjc krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.53.195@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:30:37.115939 31930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:37.109792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:36293 (local address 127.22.53.195:51747)
0926 07:30:37.110154 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:37.110171 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:37.110306 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:37.110612 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:37.110615 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:37.110853 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:37.110947 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:37.110953 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:37.111761 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:37.111766 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:37.112336 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:37.112344 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:37.112885 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:37.112888 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:37.112977 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:37.113575 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:37.113588 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:37.115759 (+ 2171us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.53.254:36293: Server oryx/127.22.53.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":98,"threads_started":1}
W20260926 07:30:37.116045 31928 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:30:37.109953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.195:51747 (local address 127.22.53.254:36293)
0926 07:30:37.110134 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:37.110139 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:37.110243 (+ 104us) server_negotiation.cc:407] Connection header received
0926 07:30:37.110432 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:37.110435 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:37.110491 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:37.110589 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:37.111078 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:37.111621 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:37.112461 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:37.112763 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:37.112876 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:37.115925 (+ 3049us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.53.195:51747: BlockingRecv error: recv got EOF from 127.22.53.195:51747 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 26 07:30:37 dist-test-slave-wdjc krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.53.193@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:30:37 dist-test-slave-wdjc krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.53.193@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:30:37.707362 31931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:37.700679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:36293 (local address 127.22.53.193:36227)
0926 07:30:37.701085 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:37.701100 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:37.701226 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:37.701491 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:37.701494 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:37.701686 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:37.701902 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:37.701911 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:37.702789 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:37.702792 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:37.703499 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:37.703513 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:37.704079 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:37.704083 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:37.704178 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:37.704957 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:37.704970 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:37.707223 (+ 2253us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.53.254:36293: Server oryx/127.22.53.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":78,"threads_started":1}
W20260926 07:30:37.707634 31932 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:30:37.700772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.193:36227 (local address 127.22.53.254:36293)
0926 07:30:37.701083 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:37.701087 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:37.701196 (+ 109us) server_negotiation.cc:407] Connection header received
0926 07:30:37.701313 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:37.701315 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:37.701362 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:37.701449 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:37.702051 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:37.702642 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:37.703661 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:37.703938 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:37.704064 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:37.707488 (+ 3424us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.53.193:36227: BlockingRecv error: recv got EOF from 127.22.53.193:36227 (error 108)
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":120,"threads_started":1}
Sep 26 07:30:37 dist-test-slave-wdjc krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.53.194@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:30:37 dist-test-slave-wdjc krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.53.194@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:30:37.918682 31933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:37.909154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:36293 (local address 127.22.53.194:60691)
0926 07:30:37.909504 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:37.909518 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:37.909626 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:37.909919 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:37.909920 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:37.910147 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:37.910246 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:37.910252 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:37.911084 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:37.911087 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:37.911684 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:37.911695 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:37.912280 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:37.912283 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:37.912355 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:37.916270 (+ 3915us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:37.916293 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:37.918543 (+ 2250us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.53.254:36293: Server oryx/127.22.53.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":126,"threads_started":1}
W20260926 07:30:37.918926 31932 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:30:37.909366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.194:60691 (local address 127.22.53.254:36293)
0926 07:30:37.909519 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:37.909524 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:37.909574 (+ 50us) server_negotiation.cc:407] Connection header received
0926 07:30:37.909734 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:37.909737 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:37.909788 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:37.909881 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:37.910395 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:37.910944 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:37.911859 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:37.912120 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:37.912222 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:37.918802 (+ 6580us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.53.194:60691: BlockingRecv error: recv got EOF from 127.22.53.194:60691 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 26 07:30:38 dist-test-slave-wdjc krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.53.195@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:30:38 dist-test-slave-wdjc krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.53.195@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:30:38.123239 31934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:38.117074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:36293 (local address 127.22.53.195:38539)
0926 07:30:38.117429 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:38.117443 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:38.117563 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:38.117835 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:38.117838 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:38.118027 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:38.118123 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:38.118128 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:38.118959 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:38.118962 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:38.119556 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:38.119566 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:38.120098 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:38.120101 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:38.120187 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:38.120847 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:38.120860 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:38.123054 (+ 2194us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.53.254:36293: Server oryx/127.22.53.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":126,"threads_started":1}
W20260926 07:30:38.123350 31932 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:30:38.117138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.195:38539 (local address 127.22.53.254:36293)
0926 07:30:38.117279 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:38.117283 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:38.117521 (+ 238us) server_negotiation.cc:407] Connection header received
0926 07:30:38.117652 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:38.117655 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:38.117705 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:38.117785 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:38.118264 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:38.118829 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:38.119709 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:38.119956 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:38.120053 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:38.123229 (+ 3176us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.53.195:38539: BlockingRecv error: recv got EOF from 127.22.53.195:38539 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 26 07:30:38 dist-test-slave-wdjc krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.53.193@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:30:38 dist-test-slave-wdjc krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.53.193@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:30:38.714668 31935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:38.708419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:36293 (local address 127.22.53.193:37893)
0926 07:30:38.708801 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:38.708819 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:38.708953 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:38.709287 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:38.709290 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:38.709530 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:38.709647 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:38.709654 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:38.710465 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:38.710470 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:38.711051 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:38.711059 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:38.711562 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:38.711565 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:38.711656 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:38.712248 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:38.712261 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:38.714483 (+ 2222us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.53.254:36293: Server oryx/127.22.53.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":151,"threads_started":1}
W20260926 07:30:38.714826 31936 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:30:38.708637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.193:37893 (local address 127.22.53.254:36293)
0926 07:30:38.708889 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:38.708893 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:38.708908 (+ 15us) server_negotiation.cc:407] Connection header received
0926 07:30:38.709079 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:38.709084 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:38.709144 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:38.709258 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:38.709776 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:38.710344 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:38.711197 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:38.711440 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:38.711554 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:38.714694 (+ 3140us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.53.193:37893: BlockingRecv error: recv got EOF from 127.22.53.193:37893 (error 108)
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":68,"threads_started":1}
Sep 26 07:30:38 dist-test-slave-wdjc krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.53.194@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:30:38 dist-test-slave-wdjc krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.53.194@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:30:38.925997 31937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:38.919573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:36293 (local address 127.22.53.194:36241)
0926 07:30:38.919901 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:38.919914 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:38.920068 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:38.920401 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:38.920404 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:38.920640 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:38.920744 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:38.920750 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:38.921705 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:38.921708 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:38.922335 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:38.922345 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:38.922855 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:38.922857 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:38.922927 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:38.923659 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:38.923671 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:38.925861 (+ 2190us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.53.254:36293: Server oryx/127.22.53.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":110,"threads_started":1}
W20260926 07:30:38.926227 31936 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:30:38.919657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.194:36241 (local address 127.22.53.254:36293)
0926 07:30:38.919828 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:38.919834 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:38.919993 (+ 159us) server_negotiation.cc:407] Connection header received
0926 07:30:38.920190 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:38.920192 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:38.920240 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:38.920324 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:38.920869 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:38.921578 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:38.922500 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:38.922734 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:38.922825 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:38.926067 (+ 3242us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.53.194:36241: BlockingRecv error: recv got EOF from 127.22.53.194:36241 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 26 07:30:39 dist-test-slave-wdjc krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.53.195@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:30:39 dist-test-slave-wdjc krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.53.195@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:30:39.132023 31938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:39.124375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:36293 (local address 127.22.53.195:55181)
0926 07:30:39.124857 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:39.124876 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:39.125002 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:39.125397 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:39.125401 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:39.125658 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:39.125787 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:39.125794 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:39.126928 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:39.126936 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:39.127588 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:39.127602 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:39.128354 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:39.128356 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:39.128452 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:39.129054 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:39.129066 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:39.131869 (+ 2803us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.53.254:36293: Server oryx/127.22.53.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":92,"threads_started":1}
W20260926 07:30:39.132416 31936 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:30:39.124556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.195:55181 (local address 127.22.53.254:36293)
0926 07:30:39.124731 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:39.124755 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:39.124941 (+ 186us) server_negotiation.cc:407] Connection header received
0926 07:30:39.125182 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:39.125185 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:39.125256 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:39.125333 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:39.125952 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:39.126546 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:39.127821 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:39.128137 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:39.128365 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:39.132274 (+ 3909us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.53.195:55181: BlockingRecv error: recv got EOF from 127.22.53.195:55181 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 26 07:30:39 dist-test-slave-wdjc krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.53.193@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:30:39 dist-test-slave-wdjc krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.53.193@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:30:39.721366 31939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:39.715641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:36293 (local address 127.22.53.193:58487)
0926 07:30:39.715940 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:39.715954 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:39.716052 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:39.716330 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:39.716332 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:39.716518 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:39.716710 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:39.716717 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:39.717497 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:39.717500 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:39.718061 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:39.718069 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:39.718521 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:39.718524 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:39.718587 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:39.719155 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:39.719167 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:39.721225 (+ 2058us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.53.254:36293: Server oryx/127.22.53.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":93,"threads_started":1}
W20260926 07:30:39.721591 31940 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:30:39.715740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.193:58487 (local address 127.22.53.254:36293)
0926 07:30:39.716010 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:39.716013 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:39.716027 (+ 14us) server_negotiation.cc:407] Connection header received
0926 07:30:39.716155 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:39.716157 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:39.716209 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:39.716288 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:39.716824 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:39.717394 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:39.718193 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:39.718416 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:39.718495 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:39.721417 (+ 2922us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.53.193:58487: BlockingRecv error: recv got EOF from 127.22.53.193:58487 (error 108)
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":98,"threads_started":1}
Sep 26 07:30:39 dist-test-slave-wdjc krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.53.194@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:30:39 dist-test-slave-wdjc krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.53.194@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:30:39.934209 31941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:39.926980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:36293 (local address 127.22.53.194:45679)
0926 07:30:39.927287 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:39.927301 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:39.927403 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:39.927702 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:39.927704 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:39.927900 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:39.928003 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:39.928009 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:39.929020 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:39.929024 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:39.929714 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:39.929748 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:39.930469 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:39.930484 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:39.930591 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:39.931466 (+ 875us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:39.931479 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:39.933982 (+ 2503us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.53.254:36293: Server oryx/127.22.53.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":112,"threads_started":1}
W20260926 07:30:39.934252 31940 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:30:39.927186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.194:45679 (local address 127.22.53.254:36293)
0926 07:30:39.927340 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:39.927344 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:39.927361 (+ 17us) server_negotiation.cc:407] Connection header received
0926 07:30:39.927502 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:39.927505 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:39.927552 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:39.927656 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:39.928139 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:39.928716 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:39.930023 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:39.930289 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:39.930385 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:39.934135 (+ 3750us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.53.194:45679: BlockingRecv error: recv got EOF from 127.22.53.194:45679 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 26 07:30:40 dist-test-slave-wdjc krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.53.195@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:30:40 dist-test-slave-wdjc krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.53.195@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:30:40.140308 31942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:40.133127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:36293 (local address 127.22.53.195:44839)
0926 07:30:40.133468 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:40.133482 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:40.133644 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:40.134027 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:40.134029 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:40.134241 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:40.134349 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:40.134354 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:40.135431 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:40.135434 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:40.136084 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:40.136095 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:40.136824 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:40.136828 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:40.136902 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:40.137707 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:40.137721 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:40.140171 (+ 2450us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.53.254:36293: Server oryx/127.22.53.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":109,"threads_started":1}
W20260926 07:30:40.140679 31940 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:30:40.133228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.195:44839 (local address 127.22.53.254:36293)
0926 07:30:40.133395 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:40.133401 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:40.133570 (+ 169us) server_negotiation.cc:407] Connection header received
0926 07:30:40.133786 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:40.133789 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:40.133856 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:40.133931 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:40.134535 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:40.135226 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:40.136263 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:40.136682 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:40.136796 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:40.140412 (+ 3616us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.53.195:44839: BlockingRecv error: recv got EOF from 127.22.53.195:44839 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 26 07:30:40 dist-test-slave-wdjc krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.53.193@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:30:40 dist-test-slave-wdjc krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.53.193@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:30:40.729218 31943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:40.722379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:36293 (local address 127.22.53.193:36035)
0926 07:30:40.722740 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:40.722760 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:40.722882 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:40.723154 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:40.723156 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:40.723394 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:40.723556 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:40.723572 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:40.724691 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:40.724695 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:40.725566 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:40.725578 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:40.726091 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:40.726095 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:40.726179 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:40.726796 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:40.726810 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:40.729010 (+ 2200us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.53.254:36293: Server oryx/127.22.53.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":129,"threads_started":1}
W20260926 07:30:40.729357 31944 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:30:40.722609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.193:36035 (local address 127.22.53.254:36293)
0926 07:30:40.722899 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:40.722903 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:40.722919 (+ 16us) server_negotiation.cc:407] Connection header received
0926 07:30:40.722964 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:40.722967 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:40.723008 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:40.723102 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:40.723747 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:40.724530 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:40.725696 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:40.725955 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:40.726085 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:40.729216 (+ 3131us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.53.193:36035: BlockingRecv error: recv got EOF from 127.22.53.193:36035 (error 108)
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":78,"threads_started":1}
Sep 26 07:30:40 dist-test-slave-wdjc krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.53.194@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:30:40 dist-test-slave-wdjc krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.53.194@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:30:40.941391 31945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:40.935349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:36293 (local address 127.22.53.194:46999)
0926 07:30:40.935794 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:40.935807 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:40.935945 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:40.936268 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:40.936271 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:40.936494 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:40.936624 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:40.936630 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:40.937428 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:40.937433 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:40.938013 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:40.938021 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:40.938500 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:40.938504 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:40.938596 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:40.939148 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:40.939159 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:40.941255 (+ 2096us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.53.254:36293: Server oryx/127.22.53.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":84,"threads_started":1}
W20260926 07:30:40.941594 31944 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:30:40.935479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.194:46999 (local address 127.22.53.254:36293)
0926 07:30:40.935747 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:40.935751 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:40.935885 (+ 134us) server_negotiation.cc:407] Connection header received
0926 07:30:40.936076 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:40.936078 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:40.936124 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:40.936245 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:40.936757 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:40.937312 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:40.938146 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:40.938376 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:40.938482 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:40.941425 (+ 2943us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.53.194:46999: BlockingRecv error: recv got EOF from 127.22.53.194:46999 (error 108)
Metrics: {"server-negotiator.queue_time_us":164}
Sep 26 07:30:41 dist-test-slave-wdjc krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.53.195@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:30:41 dist-test-slave-wdjc krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.53.195@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:30:41.147930 31946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:41.141498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:36293 (local address 127.22.53.195:34533)
0926 07:30:41.142011 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:41.142031 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:41.142151 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:41.142571 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:41.142575 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:41.142827 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:41.142960 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:41.142968 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:41.143782 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:41.143785 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:41.144385 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:41.144394 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:41.144909 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:41.144938 (+ 29us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:41.145024 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:41.145615 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:41.145627 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:41.147782 (+ 2155us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.53.254:36293: Server oryx/127.22.53.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":121,"threads_started":1}
W20260926 07:30:41.148110 31944 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:30:41.141721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.195:34533 (local address 127.22.53.254:36293)
0926 07:30:41.141879 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:41.141886 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:41.142122 (+ 236us) server_negotiation.cc:407] Connection header received
0926 07:30:41.142329 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:41.142333 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:41.142397 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:41.142513 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:41.143115 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:41.143669 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:41.144519 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:41.144785 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:41.144874 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:41.147969 (+ 3095us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.53.195:34533: BlockingRecv error: recv got EOF from 127.22.53.195:34533 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 26 07:30:41 dist-test-slave-wdjc krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.53.193@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:30:41 dist-test-slave-wdjc krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.53.193@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:30:41.736599 31947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:41.730178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:36293 (local address 127.22.53.193:58413)
0926 07:30:41.730508 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:41.730534 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:41.730648 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:41.730910 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:41.730912 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:41.731103 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:41.731202 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:41.731207 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:41.732026 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:41.732029 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:41.732766 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:41.732779 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:41.733360 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:41.733364 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:41.733454 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:41.734122 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:41.734135 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:41.736395 (+ 2260us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.53.254:36293: Server oryx/127.22.53.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":93,"threads_started":1}
W20260926 07:30:41.736848 31948 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:30:41.730357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.193:58413 (local address 127.22.53.254:36293)
0926 07:30:41.730629 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:41.730634 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:41.730650 (+ 16us) server_negotiation.cc:407] Connection header received
0926 07:30:41.730718 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:41.730720 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:41.730768 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:41.730850 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:41.731336 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:41.731892 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:41.732936 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:41.733190 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:41.733314 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:41.736671 (+ 3357us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.53.193:58413: BlockingRecv error: recv got EOF from 127.22.53.193:58413 (error 108)
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":82,"threads_started":1}
Sep 26 07:30:41 dist-test-slave-wdjc krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.53.194@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:30:41 dist-test-slave-wdjc krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.53.194@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:30:41.949851 31949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:41.942471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:36293 (local address 127.22.53.194:34445)
0926 07:30:41.943020 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:41.943041 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:41.943217 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:41.943514 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:41.943517 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:41.943816 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:41.943929 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:41.943935 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:41.945093 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:41.945098 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:41.945691 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:41.945701 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:41.946255 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:41.946259 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:41.946352 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:41.947047 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:41.947075 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:41.949714 (+ 2639us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.53.254:36293: Server oryx/127.22.53.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":282,"threads_started":1}
W20260926 07:30:41.950138 31948 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:30:41.942607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.194:34445 (local address 127.22.53.254:36293)
0926 07:30:41.942969 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:41.942975 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:41.943144 (+ 169us) server_negotiation.cc:407] Connection header received
0926 07:30:41.943332 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:41.943336 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:41.943393 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:41.943492 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:41.944073 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:41.944955 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:41.945831 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:41.946107 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:41.946228 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:41.949962 (+ 3734us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.53.194:34445: BlockingRecv error: recv got EOF from 127.22.53.194:34445 (error 108)
Metrics: {"server-negotiator.queue_time_us":217}
Sep 26 07:30:42 dist-test-slave-wdjc krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.53.195@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:30:42 dist-test-slave-wdjc krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.53.195@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:30:42.154939 31950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:42.148935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:36293 (local address 127.22.53.195:38765)
0926 07:30:42.149238 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:42.149252 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:42.149353 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:42.149693 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:42.149695 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:42.149886 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:42.149983 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:42.149988 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:42.150885 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:42.150889 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:42.151543 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:42.151553 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:42.152048 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:42.152052 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:42.152121 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:42.152781 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:42.152793 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:42.154808 (+ 2015us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.53.254:36293: Server oryx/127.22.53.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":125,"threads_started":1}
W20260926 07:30:42.155212 31948 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:30:42.149099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.195:38765 (local address 127.22.53.254:36293)
0926 07:30:42.149236 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:42.149240 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:42.149346 (+ 106us) server_negotiation.cc:407] Connection header received
0926 07:30:42.149439 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:42.149442 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:42.149493 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:42.149582 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:42.150123 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:42.150749 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:42.151680 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:42.151911 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:42.152011 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:42.155082 (+ 3071us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.53.195:38765: BlockingRecv error: recv got EOF from 127.22.53.195:38765 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 26 07:30:42 dist-test-slave-wdjc krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.53.193@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:30:42 dist-test-slave-wdjc krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.53.193@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:30:42.744751 31951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:42.737764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:36293 (local address 127.22.53.193:50801)
0926 07:30:42.738169 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:42.738201 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:42.738317 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:42.738731 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:42.738734 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:42.738929 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:42.739040 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:42.739057 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:42.740009 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:42.740012 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:42.740698 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:42.740709 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:42.741243 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:42.741246 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:42.741317 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:42.741985 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:42.741998 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:42.744622 (+ 2624us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.53.254:36293: Server oryx/127.22.53.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":129,"threads_started":1}
W20260926 07:30:42.744918 31952 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:30:42.737976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.193:50801 (local address 127.22.53.254:36293)
0926 07:30:42.738322 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:42.738326 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:42.738347 (+ 21us) server_negotiation.cc:407] Connection header received
0926 07:30:42.738398 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:42.738401 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:42.738451 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:42.738575 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:42.739185 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:42.739854 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:42.740844 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:42.741109 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:42.741196 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:42.744775 (+ 3579us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.53.193:50801: BlockingRecv error: recv got EOF from 127.22.53.193:50801 (error 108)
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":68,"threads_started":1}
Sep 26 07:30:42 dist-test-slave-wdjc krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.53.194@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:30:42 dist-test-slave-wdjc krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.53.194@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:30:42.960887 31953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:42.950955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:36293 (local address 127.22.53.194:58321)
0926 07:30:42.951290 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:42.951304 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:42.951421 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:42.951730 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:42.951734 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:42.951971 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:42.952113 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:42.952120 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:42.953145 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:42.953149 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:42.953760 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:42.953770 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:42.954208 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:42.954211 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:42.954275 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:42.958365 (+ 4090us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:42.958383 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:42.960751 (+ 2368us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.53.254:36293: Server oryx/127.22.53.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":94,"threads_started":1}
W20260926 07:30:42.961110 31952 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:30:42.951146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.194:58321 (local address 127.22.53.254:36293)
0926 07:30:42.951319 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:42.951323 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:42.951364 (+ 41us) server_negotiation.cc:407] Connection header received
0926 07:30:42.951515 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:42.951517 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:42.951578 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:42.951665 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:42.952284 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:42.953027 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:42.953871 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:42.954093 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:42.954178 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:42.960962 (+ 6784us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.53.194:58321: BlockingRecv error: recv got EOF from 127.22.53.194:58321 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 26 07:30:43 dist-test-slave-wdjc krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.53.195@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:30:43 dist-test-slave-wdjc krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.53.195@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:30:43.162617 31954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:43.156128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:36293 (local address 127.22.53.195:46737)
0926 07:30:43.156454 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:43.156468 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:43.156750 (+ 282us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:43.157051 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:43.157054 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:43.157248 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:43.157349 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:43.157354 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:43.158221 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:43.158230 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:43.158827 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:43.158835 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:43.159360 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:43.159364 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:43.159454 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:43.160047 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:43.160061 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:43.162421 (+ 2360us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.53.254:36293: Server oryx/127.22.53.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":93,"threads_started":1}
W20260926 07:30:43.162729 31952 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:30:43.156133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.195:46737 (local address 127.22.53.254:36293)
0926 07:30:43.156306 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:43.156311 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:43.156689 (+ 378us) server_negotiation.cc:407] Connection header received
0926 07:30:43.156861 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:43.156864 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:43.156917 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:43.156996 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:43.157456 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:43.158100 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:43.158964 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:43.159226 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:43.159348 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:43.162606 (+ 3258us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.53.195:46737: BlockingRecv error: recv got EOF from 127.22.53.195:46737 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 26 07:30:43 dist-test-slave-wdjc krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.53.193@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:30:43 dist-test-slave-wdjc krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.53.193@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:30:43.751479 31955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:43.745600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:36293 (local address 127.22.53.193:36595)
0926 07:30:43.745875 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:43.745888 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:43.745990 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:43.746244 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:43.746246 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:43.746431 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:43.746528 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:43.746533 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:43.747305 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:43.747307 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:43.747860 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:43.747867 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:43.748466 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:43.748470 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:43.748615 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:43.749251 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:43.749271 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:43.751343 (+ 2072us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.53.254:36293: Server oryx/127.22.53.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":83,"threads_started":1}
W20260926 07:30:43.751757 31956 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:30:43.745709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.193:36595 (local address 127.22.53.254:36293)
0926 07:30:43.745939 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:43.745943 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:43.745955 (+ 12us) server_negotiation.cc:407] Connection header received
0926 07:30:43.746083 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:43.746085 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:43.746128 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:43.746197 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:43.746656 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:43.747203 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:43.748013 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:43.748292 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:43.748411 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:43.751564 (+ 3153us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.53.193:36595: BlockingRecv error: recv got EOF from 127.22.53.193:36595 (error 108)
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":59,"threads_started":1}
Sep 26 07:30:43 dist-test-slave-wdjc krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.53.194@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:30:43 dist-test-slave-wdjc krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.53.194@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:30:43.968787 31957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:43.961983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:36293 (local address 127.22.53.194:39261)
0926 07:30:43.962263 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:43.962284 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:43.962426 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:43.962720 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:43.962723 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:43.962960 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:43.963055 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:43.963060 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:43.963897 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:43.963903 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:43.964498 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:43.964506 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:43.965260 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:43.965264 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:43.965362 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:43.966126 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:43.966138 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:43.968646 (+ 2508us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.53.254:36293: Server oryx/127.22.53.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":104,"threads_started":1}
W20260926 07:30:43.968922 31956 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:30:43.962279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.194:39261 (local address 127.22.53.254:36293)
0926 07:30:43.962459 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:43.962466 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:43.962505 (+ 39us) server_negotiation.cc:407] Connection header received
0926 07:30:43.962563 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:43.962566 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:43.962609 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:43.962702 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:43.963199 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:43.963764 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:43.964837 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:43.965111 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:43.965250 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:43.968791 (+ 3541us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.53.194:39261: BlockingRecv error: recv got EOF from 127.22.53.194:39261 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 26 07:30:44 dist-test-slave-wdjc krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.53.195@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:30:44 dist-test-slave-wdjc krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.53.195@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:30:44.170001 31958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:44.163515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:36293 (local address 127.22.53.195:59477)
0926 07:30:44.164062 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:44.164076 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:44.164225 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:44.164514 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:44.164517 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:44.164765 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:44.164860 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:44.164866 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:44.165707 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:44.165713 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:44.166299 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:44.166307 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:44.166830 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:44.166834 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:44.166932 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:44.167617 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:44.167631 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:44.169849 (+ 2218us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.53.254:36293: Server oryx/127.22.53.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":105,"threads_started":1}
W20260926 07:30:44.170262 31956 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:30:44.163591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.195:59477 (local address 127.22.53.254:36293)
0926 07:30:44.163794 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:44.163798 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:44.164175 (+ 377us) server_negotiation.cc:407] Connection header received
0926 07:30:44.164341 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:44.164344 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:44.164405 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:44.164493 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:44.165009 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:44.165582 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:44.166443 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:44.166695 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:44.166813 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:44.170080 (+ 3267us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.53.195:59477: BlockingRecv error: recv got EOF from 127.22.53.195:59477 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Sep 26 07:30:44 dist-test-slave-wdjc krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.53.193@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:30:44 dist-test-slave-wdjc krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.53.193@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:30:44.759023 31959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:44.752427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:36293 (local address 127.22.53.193:39611)
0926 07:30:44.752845 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:44.752907 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:44.753014 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:44.753258 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:44.753261 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:44.753477 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:44.753580 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:44.753586 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:44.754468 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:44.754472 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:44.755124 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:44.755136 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:44.755743 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:44.755746 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:44.755820 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:44.756499 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:44.756514 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:44.758834 (+ 2320us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.53.254:36293: Server oryx/127.22.53.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":97,"threads_started":1}
W20260926 07:30:44.759240 31960 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:30:44.752635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.193:39611 (local address 127.22.53.254:36293)
0926 07:30:44.753009 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:44.753013 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:44.753029 (+ 16us) server_negotiation.cc:407] Connection header received
0926 07:30:44.753084 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:44.753087 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:44.753130 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:44.753211 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:44.753751 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:44.754330 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:44.755335 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:44.755592 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:44.755688 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:44.759060 (+ 3372us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.53.193:39611: BlockingRecv error: recv got EOF from 127.22.53.193:39611 (error 108)
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":106,"threads_started":1}
Sep 26 07:30:44 dist-test-slave-wdjc krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.53.194@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:30:44 dist-test-slave-wdjc krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.53.194@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:30:44.975773 31961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:44.969720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:36293 (local address 127.22.53.194:52757)
0926 07:30:44.970013 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:44.970030 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:44.970249 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:44.970535 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:44.970538 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:44.970754 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:44.970846 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:44.970852 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:44.971643 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:44.971648 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:44.972223 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:44.972232 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:44.972792 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:44.972796 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:44.972882 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:44.973457 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:44.973469 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:44.975581 (+ 2112us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.53.254:36293: Server oryx/127.22.53.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":82,"threads_started":1}
W20260926 07:30:44.975864 31960 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:30:44.969803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.194:52757 (local address 127.22.53.254:36293)
0926 07:30:44.969983 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:44.969989 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:44.970150 (+ 161us) server_negotiation.cc:407] Connection header received
0926 07:30:44.970359 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:44.970362 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:44.970420 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:44.970511 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:44.970966 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:44.971513 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:44.972351 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:44.972672 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:44.972783 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:44.975740 (+ 2957us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.53.194:52757: BlockingRecv error: recv got EOF from 127.22.53.194:52757 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 26 07:30:45 dist-test-slave-wdjc krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.53.195@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:30:45 dist-test-slave-wdjc krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.53.195@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:30:45.177917 31962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:45.171178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:36293 (local address 127.22.53.195:36741)
0926 07:30:45.171516 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:45.171529 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:45.171648 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:45.172173 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:45.172176 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:45.172367 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:45.172479 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:45.172484 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:45.173360 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:45.173363 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:45.174006 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:45.174016 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:45.174631 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:45.174635 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:45.174745 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:45.175379 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:45.175394 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:45.177724 (+ 2330us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.53.254:36293: Server oryx/127.22.53.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":151,"threads_started":1}
W20260926 07:30:45.178026 31960 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:30:45.171308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.195:36741 (local address 127.22.53.254:36293)
0926 07:30:45.171465 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:45.171469 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:45.171618 (+ 149us) server_negotiation.cc:407] Connection header received
0926 07:30:45.171808 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:45.171812 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:45.171926 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:45.172141 (+ 215us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:45.172647 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:45.173242 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:45.174174 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:45.174504 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:45.174612 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:45.177884 (+ 3272us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.53.195:36741: BlockingRecv error: recv got EOF from 127.22.53.195:36741 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 26 07:30:45 dist-test-slave-wdjc krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.53.193@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:30:45 dist-test-slave-wdjc krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.53.193@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:30:45.766528 31963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:45.760059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:36293 (local address 127.22.53.193:59581)
0926 07:30:45.760379 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:45.760408 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:45.760516 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:45.760887 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:45.760890 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:45.761126 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:45.761239 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:45.761245 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:45.762150 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:45.762152 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:45.762755 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:45.762765 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:45.763292 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:45.763295 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:45.763365 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:45.763936 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:45.763949 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:45.766302 (+ 2353us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.53.254:36293: Server oryx/127.22.53.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":117,"threads_started":1}
W20260926 07:30:45.766620 31964 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:30:45.760279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.193:59581 (local address 127.22.53.254:36293)
0926 07:30:45.760522 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:45.760526 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:45.760613 (+ 87us) server_negotiation.cc:407] Connection header received
0926 07:30:45.760664 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:45.760666 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:45.760712 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:45.760798 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:45.761422 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:45.762012 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:45.762899 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:45.763167 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:45.763267 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:45.766477 (+ 3210us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.53.193:59581: BlockingRecv error: recv got EOF from 127.22.53.193:59581 (error 108)
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":89,"threads_started":1}
Sep 26 07:30:45 dist-test-slave-wdjc krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.53.194@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:30:45 dist-test-slave-wdjc krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.53.194@KRBTEST.COM for oryx/127.22.53.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:30:45.986699 31965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:45.977129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:36293 (local address 127.22.53.194:54695)
0926 07:30:45.977403 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:45.977417 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:45.977509 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:45.978058 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:45.978060 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:45.978248 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:45.978346 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:45.978352 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:45.979159 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:45.979162 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:45.979866 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:45.979877 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:45.980492 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:45.980495 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:45.980630 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:45.984518 (+ 3888us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:45.984592 (+ 74us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:45.986554 (+ 1962us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.53.254:36293: Server oryx/127.22.53.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":100,"threads_started":1}
W20260926 07:30:45.986949 31964 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:30:45.977242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.194:54695 (local address 127.22.53.254:36293)
0926 07:30:45.977400 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:45.977404 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:45.977705 (+ 301us) server_negotiation.cc:407] Connection header received
0926 07:30:45.977785 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:45.977787 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:45.977836 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:45.977978 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:45.978490 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:45.979031 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:45.980011 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:45.980360 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:45.980466 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:45.986819 (+ 6353us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.53.194:54695: BlockingRecv error: recv got EOF from 127.22.53.194:54695 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 07:30:46.054045 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 31489
I20260926 07:30:46.060199 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 31641
I20260926 07:30:46.065776 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 31773
I20260926 07:30:46.071609 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 31419
2026-09-26T07:30:46Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18203 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-26T07:30:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:30:46Z Disabled control of system clock
I20260926 07:30:46.104202 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407783529717-22743-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407783529717-22743-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407783529717-22743-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.53.254:40197
--webserver_interface=127.22.53.254
--webserver_port=0
--builtin_ntp_servers=127.22.53.212:39697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.53.254:40197
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 07:30:46.250737 31969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:30:46.251084 31969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:30:46.251125 31969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:30:46.255622 31969 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:30:46.255726 31969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:30:46.255743 31969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:30:46.255760 31969 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:30:46.255777 31969 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:30:46.261287 31969 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:39697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407783529717-22743-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407783529717-22743-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.53.254:40197
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.53.254:40197
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.53.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31969
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407783529717-22743-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:30:46.262549 31969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:30:46.265405 31969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:30:46.277688 31974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:46.277824 31975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:46.277807 31969 server_base.cc:1034] running on GCE node
W20260926 07:30:46.277688 31977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:46.278710 31969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:30:46.280138 31969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:30:46.283615 31969 hybrid_clock.cc:648] HybridClock initialized: now 1790407846283559 us; error 77 us; skew 500 ppm
I20260926 07:30:46.286950 31969 webserver.cc:460] Webserver started at http://127.22.53.254:33931/ using document root <none> and password file <none>
I20260926 07:30:46.287772 31969 fs_manager.cc:359] Metadata directory not provided
I20260926 07:30:46.287859 31969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:30:46.288216 31969 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:30:46.290934 31969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407783529717-22743-0/minicluster-data/master-0/data/instance:
uuid: "2dff5171a4ed4cc58a749033919d5bb9"
format_stamp: "Formatted at 2026-09-26 07:30:46 on dist-test-slave-wdjc"
server_key: "e7be4bc15857559d4b3364ecf51dce8a"
server_key_iv: "16c46d86666e47be0d2eb0b020387b78"
server_key_version: "encryptionkey@0"
I20260926 07:30:46.292032 31969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407783529717-22743-0/minicluster-data/master-0/wal/instance:
uuid: "2dff5171a4ed4cc58a749033919d5bb9"
format_stamp: "Formatted at 2026-09-26 07:30:46 on dist-test-slave-wdjc"
server_key: "e7be4bc15857559d4b3364ecf51dce8a"
server_key_iv: "16c46d86666e47be0d2eb0b020387b78"
server_key_version: "encryptionkey@0"
I20260926 07:30:46.298794 31969 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.009s sys 0.000s
I20260926 07:30:46.302440 31983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:46.305142 31969 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 07:30:46.305446 31969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407783529717-22743-0/minicluster-data/master-0/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407783529717-22743-0/minicluster-data/master-0/wal
uuid: "2dff5171a4ed4cc58a749033919d5bb9"
format_stamp: "Formatted at 2026-09-26 07:30:46 on dist-test-slave-wdjc"
server_key: "e7be4bc15857559d4b3364ecf51dce8a"
server_key_iv: "16c46d86666e47be0d2eb0b020387b78"
server_key_version: "encryptionkey@0"
I20260926 07:30:46.305941 31969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407783529717-22743-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407783529717-22743-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407783529717-22743-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:30:46.332733 31969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:30:46.333637 31969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:30:46.333918 31969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:30:46.344385 31969 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.254:40197
I20260926 07:30:46.344405 32035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.254:40197 every 8 connection(s)
I20260926 07:30:46.345597 31969 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
I20260926 07:30:46.349630 32036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:30:46.352866 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 31969
I20260926 07:30:46.353006 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407783529717-22743-0/minicluster-data/master-0/wal/instance
I20260926 07:30:46.353579 22743 external_mini_cluster.cc:1428] Setting key cd9461eb727d7fb761194ec6df37e4a0
I20260926 07:30:46.358232 32036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2dff5171a4ed4cc58a749033919d5bb9: Bootstrap starting.
I20260926 07:30:46.361707 32036 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2dff5171a4ed4cc58a749033919d5bb9: Neither blocks nor log segments found. Creating new log.
I20260926 07:30:46.362637 32036 log.cc:824] T 00000000000000000000000000000000 P 2dff5171a4ed4cc58a749033919d5bb9: Log is configured to *not* fsync() on all Append() calls
I20260926 07:30:46.365428 32036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2dff5171a4ed4cc58a749033919d5bb9: No bootstrap required, opened a new log
I20260926 07:30:46.366257 32039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:46.355547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38364 (local address 127.22.53.254:40197)
0926 07:30:46.356524 (+ 977us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:46.356628 (+ 104us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:46.356675 (+ 47us) server_negotiation.cc:407] Connection header received
0926 07:30:46.357449 (+ 774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:46.357471 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:46.358006 (+ 535us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:46.358456 (+ 450us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:46.359161 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:46.360519 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:46.361752 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:46.362434 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:46.362678 (+ 244us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:46.363792 (+ 1114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:46.363798 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:46.363843 (+ 45us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:30:46.364326 (+ 483us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:46.364398 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:30:46.364423 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:46.364766 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:46.364993 (+ 227us) server_negotiation.cc:299] Negotiation successful
0926 07:30:46.365370 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":206,"threads_started":1}
I20260926 07:30:46.370931 32036 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2dff5171a4ed4cc58a749033919d5bb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dff5171a4ed4cc58a749033919d5bb9" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 40197 } }
I20260926 07:30:46.371253 32036 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2dff5171a4ed4cc58a749033919d5bb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:30:46.371294 32036 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2dff5171a4ed4cc58a749033919d5bb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2dff5171a4ed4cc58a749033919d5bb9, State: Initialized, Role: FOLLOWER
I20260926 07:30:46.372278 32036 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2dff5171a4ed4cc58a749033919d5bb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dff5171a4ed4cc58a749033919d5bb9" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 40197 } }
I20260926 07:30:46.372723 32036 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2dff5171a4ed4cc58a749033919d5bb9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:30:46.372900 32036 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2dff5171a4ed4cc58a749033919d5bb9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:30:46.373064 32036 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2dff5171a4ed4cc58a749033919d5bb9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:30:46.374378 32036 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2dff5171a4ed4cc58a749033919d5bb9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dff5171a4ed4cc58a749033919d5bb9" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 40197 } }
I20260926 07:30:46.374924 32036 leader_election.cc:302] T 00000000000000000000000000000000 P 2dff5171a4ed4cc58a749033919d5bb9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2dff5171a4ed4cc58a749033919d5bb9; no voters:
I20260926 07:30:46.375440 32036 leader_election.cc:288] T 00000000000000000000000000000000 P 2dff5171a4ed4cc58a749033919d5bb9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:30:46.375584 32041 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2dff5171a4ed4cc58a749033919d5bb9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:30:46.375881 32041 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2dff5171a4ed4cc58a749033919d5bb9 [term 1 LEADER]: Becoming Leader. State: Replica: 2dff5171a4ed4cc58a749033919d5bb9, State: Running, Role: LEADER
I20260926 07:30:46.376405 32041 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2dff5171a4ed4cc58a749033919d5bb9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dff5171a4ed4cc58a749033919d5bb9" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 40197 } }
I20260926 07:30:46.377131 32036 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2dff5171a4ed4cc58a749033919d5bb9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:30:46.378743 32043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2dff5171a4ed4cc58a749033919d5bb9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2dff5171a4ed4cc58a749033919d5bb9. Latest consensus state: current_term: 1 leader_uuid: "2dff5171a4ed4cc58a749033919d5bb9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dff5171a4ed4cc58a749033919d5bb9" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 40197 } } }
I20260926 07:30:46.378996 32043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2dff5171a4ed4cc58a749033919d5bb9 [sys.catalog]: This master's current role is: LEADER
I20260926 07:30:46.378743 32042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2dff5171a4ed4cc58a749033919d5bb9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2dff5171a4ed4cc58a749033919d5bb9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dff5171a4ed4cc58a749033919d5bb9" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 40197 } } }
I20260926 07:30:46.380141 32042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2dff5171a4ed4cc58a749033919d5bb9 [sys.catalog]: This master's current role is: LEADER
I20260926 07:30:46.380414 32052 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:30:46.383682 32052 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:30:46.391629 32052 catalog_manager.cc:1347] Generated new cluster ID: 37a2842dc6454502936f792d36023a52
I20260926 07:30:46.391741 32052 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:30:46.405475 32052 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:30:46.406662 32052 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:30:46.417654 32052 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2dff5171a4ed4cc58a749033919d5bb9: Generated new TSK 0
I20260926 07:30:46.418599 32052 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:30:46.433282 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407783529717-22743-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407783529717-22743-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407783529717-22743-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407783529717-22743-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.53.193:0
--local_ip_for_outbound_sockets=127.22.53.193
--webserver_interface=127.22.53.193
--webserver_port=0
--tserver_master_addrs=127.22.53.254:40197
--builtin_ntp_servers=127.22.53.212:39697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260926 07:30:46.566305 32060 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:30:46.566684 32060 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:30:46.566764 32060 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:30:46.571291 32060 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:30:46.571398 32060 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:30:46.571499 32060 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.53.193
I20260926 07:30:46.576817 32060 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:39697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407783529717-22743-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407783529717-22743-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.53.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407783529717-22743-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.53.193
--webserver_port=0
--tserver_master_addrs=127.22.53.254:40197
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32060
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.53.193
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407783529717-22743-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:30:46.578224 32060 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:30:46.580130 32060 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:30:46.593076 32066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:46.593142 32060 server_base.cc:1034] running on GCE node
W20260926 07:30:46.593083 32065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:46.593433 32068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:46.593920 32060 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:30:46.594763 32060 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:30:46.598140 32060 hybrid_clock.cc:648] HybridClock initialized: now 1790407846598107 us; error 64 us; skew 500 ppm
I20260926 07:30:46.600497 32060 webserver.cc:460] Webserver started at http://127.22.53.193:44271/ using document root <none> and password file <none>
I20260926 07:30:46.601301 32060 fs_manager.cc:359] Metadata directory not provided
I20260926 07:30:46.601382 32060 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:30:46.601629 32060 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:30:46.604399 32060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407783529717-22743-0/minicluster-data/ts-0/data/instance:
uuid: "e1b08677014143d087660e4da60640be"
format_stamp: "Formatted at 2026-09-26 07:30:46 on dist-test-slave-wdjc"
server_key: "7854b1cbd078d7420a8580872d23b38f"
server_key_iv: "e5dc4384b1151b7c5d32bb2a0fa81c74"
server_key_version: "encryptionkey@0"
I20260926 07:30:46.605188 32060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407783529717-22743-0/minicluster-data/ts-0/wal/instance:
uuid: "e1b08677014143d087660e4da60640be"
format_stamp: "Formatted at 2026-09-26 07:30:46 on dist-test-slave-wdjc"
server_key: "7854b1cbd078d7420a8580872d23b38f"
server_key_iv: "e5dc4384b1151b7c5d32bb2a0fa81c74"
server_key_version: "encryptionkey@0"
I20260926 07:30:46.610198 32060 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260926 07:30:46.613816 32074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:46.615499 32060 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.001s
I20260926 07:30:46.615692 32060 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407783529717-22743-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407783529717-22743-0/minicluster-data/ts-0/wal
uuid: "e1b08677014143d087660e4da60640be"
format_stamp: "Formatted at 2026-09-26 07:30:46 on dist-test-slave-wdjc"
server_key: "7854b1cbd078d7420a8580872d23b38f"
server_key_iv: "e5dc4384b1151b7c5d32bb2a0fa81c74"
server_key_version: "encryptionkey@0"
I20260926 07:30:46.615849 32060 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407783529717-22743-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407783529717-22743-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407783529717-22743-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:30:46.647351 32060 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:30:46.648173 32060 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:30:46.648401 32060 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:30:46.649138 32060 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:30:46.650151 32060 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:30:46.650226 32060 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:46.650301 32060 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:30:46.650355 32060 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:46.660269 32060 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.193:35925
I20260926 07:30:46.660292 32187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.193:35925 every 8 connection(s)
I20260926 07:30:46.661394 32060 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407783529717-22743-0/minicluster-data/ts-0/data/info.pb
I20260926 07:30:46.662326 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 32060
I20260926 07:30:46.662459 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407783529717-22743-0/minicluster-data/ts-0/wal/instance
I20260926 07:30:46.662833 22743 external_mini_cluster.cc:1428] Setting key 527e9be1fa52fd6820afaaad070999a5
I20260926 07:30:46.665086 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407783529717-22743-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407783529717-22743-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407783529717-22743-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407783529717-22743-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.53.194:0
--local_ip_for_outbound_sockets=127.22.53.194
--webserver_interface=127.22.53.194
--webserver_port=0
--tserver_master_addrs=127.22.53.254:40197
--builtin_ntp_servers=127.22.53.212:39697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260926 07:30:46.673972 32039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:46.663733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.193:53925 (local address 127.22.53.254:40197)
0926 07:30:46.663870 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:46.663882 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:46.667012 (+ 3130us) server_negotiation.cc:407] Connection header received
0926 07:30:46.668567 (+ 1555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:46.668571 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:46.668638 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:46.668722 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:46.669781 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:46.670447 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:46.671721 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:46.671972 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:46.672097 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:46.673411 (+ 1314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:46.673423 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:46.673426 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:30:46.673452 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:46.673471 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:30:46.673479 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:46.673561 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:46.673739 (+ 178us) server_negotiation.cc:299] Negotiation successful
0926 07:30:46.673812 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 07:30:46.674984 32190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:46.664259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:40197 (local address 127.22.53.193:53925)
0926 07:30:46.666803 (+ 2544us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:46.666855 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:46.668278 (+ 1423us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:46.668944 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:46.668953 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:46.669057 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:46.669609 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:46.669622 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:46.670599 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:46.670607 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:46.671573 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:46.671584 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:46.672155 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:46.672160 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:46.672347 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:46.673097 (+ 750us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:30:46.673099 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:46.673147 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:30:46.673224 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:30:46.673576 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:30:46.673588 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:30:46.673600 (+ 12us) client_negotiation.cc:769] Sending connection context
0926 07:30:46.673739 (+ 139us) client_negotiation.cc:240] Negotiation successful
0926 07:30:46.674094 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2073,"thread_start_us":134,"threads_started":1}
I20260926 07:30:46.676524 32188 heartbeater.cc:347] Connected to a master server at 127.22.53.254:40197
I20260926 07:30:46.676862 32188 heartbeater.cc:464] Registering TS with master...
I20260926 07:30:46.677572 32188 heartbeater.cc:511] Master 127.22.53.254:40197 requested a full tablet report, sending...
I20260926 07:30:46.679705 32000 ts_manager.cc:194] Registered new tserver with Master: e1b08677014143d087660e4da60640be (127.22.53.193:35925)
I20260926 07:30:46.681535 32000 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.53.193:53925
W20260926 07:30:46.791416 32191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:30:46.791733 32191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:30:46.791774 32191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:30:46.796269 32191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:30:46.796361 32191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:30:46.796422 32191 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.53.194
I20260926 07:30:46.802068 32191 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:39697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407783529717-22743-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407783529717-22743-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.53.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407783529717-22743-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.53.194
--webserver_port=0
--tserver_master_addrs=127.22.53.254:40197
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32191
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.53.194
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407783529717-22743-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:30:46.803264 32191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:30:46.804847 32191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:30:46.816174 32199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:46.816164 32196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:46.816167 32197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:46.816593 32191 server_base.cc:1034] running on GCE node
I20260926 07:30:46.817175 32191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:30:46.817925 32191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:30:46.819139 32191 hybrid_clock.cc:648] HybridClock initialized: now 1790407846819100 us; error 56 us; skew 500 ppm
I20260926 07:30:46.821444 32191 webserver.cc:460] Webserver started at http://127.22.53.194:39453/ using document root <none> and password file <none>
I20260926 07:30:46.822116 32191 fs_manager.cc:359] Metadata directory not provided
I20260926 07:30:46.822196 32191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:30:46.822445 32191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:30:46.824904 32191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407783529717-22743-0/minicluster-data/ts-1/data/instance:
uuid: "af48d8b8569b4deea88e325391785af6"
format_stamp: "Formatted at 2026-09-26 07:30:46 on dist-test-slave-wdjc"
server_key: "fdc30e1322028872e797c68d53f30f2a"
server_key_iv: "6d53fe70ee7f53a53ee14f9c9e191063"
server_key_version: "encryptionkey@0"
I20260926 07:30:46.825605 32191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407783529717-22743-0/minicluster-data/ts-1/wal/instance:
uuid: "af48d8b8569b4deea88e325391785af6"
format_stamp: "Formatted at 2026-09-26 07:30:46 on dist-test-slave-wdjc"
server_key: "fdc30e1322028872e797c68d53f30f2a"
server_key_iv: "6d53fe70ee7f53a53ee14f9c9e191063"
server_key_version: "encryptionkey@0"
I20260926 07:30:46.830003 32191 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260926 07:30:46.833220 32205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:46.834899 32191 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 07:30:46.835085 32191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407783529717-22743-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407783529717-22743-0/minicluster-data/ts-1/wal
uuid: "af48d8b8569b4deea88e325391785af6"
format_stamp: "Formatted at 2026-09-26 07:30:46 on dist-test-slave-wdjc"
server_key: "fdc30e1322028872e797c68d53f30f2a"
server_key_iv: "6d53fe70ee7f53a53ee14f9c9e191063"
server_key_version: "encryptionkey@0"
I20260926 07:30:46.835240 32191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407783529717-22743-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407783529717-22743-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407783529717-22743-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:30:46.852228 32191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:30:46.853140 32191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:30:46.853389 32191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:30:46.854071 32191 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:30:46.855167 32191 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:30:46.855250 32191 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:46.855329 32191 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:30:46.855371 32191 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:46.865818 32191 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.194:34177
I20260926 07:30:46.865846 32318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.194:34177 every 8 connection(s)
I20260926 07:30:46.866887 32191 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407783529717-22743-0/minicluster-data/ts-1/data/info.pb
I20260926 07:30:46.875047 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 32191
I20260926 07:30:46.875176 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407783529717-22743-0/minicluster-data/ts-1/wal/instance
I20260926 07:30:46.875483 22743 external_mini_cluster.cc:1428] Setting key d7e924390828a258cdbdeca779d92500
I20260926 07:30:46.877976 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407783529717-22743-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407783529717-22743-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407783529717-22743-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407783529717-22743-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.53.195:0
--local_ip_for_outbound_sockets=127.22.53.195
--webserver_interface=127.22.53.195
--webserver_port=0
--tserver_master_addrs=127.22.53.254:40197
--builtin_ntp_servers=127.22.53.212:39697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260926 07:30:46.878747 32039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:46.869594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.194:59645 (local address 127.22.53.254:40197)
0926 07:30:46.869754 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:46.869758 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:46.871036 (+ 1278us) server_negotiation.cc:407] Connection header received
0926 07:30:46.872521 (+ 1485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:46.872524 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:46.872659 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:46.872751 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:46.873779 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:46.874362 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:46.876136 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:46.876403 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:46.876518 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:46.878052 (+ 1534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:46.878057 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:46.878060 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:30:46.878075 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:46.878099 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:30:46.878110 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:46.878192 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:46.878424 (+ 232us) server_negotiation.cc:299] Negotiation successful
0926 07:30:46.878510 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 07:30:46.879832 32321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:46.870020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:40197 (local address 127.22.53.194:59645)
0926 07:30:46.870853 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:46.870901 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:46.872279 (+ 1378us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:46.872933 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:46.872942 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:46.873037 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:46.873605 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:46.873618 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:46.874517 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:46.874520 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:46.875996 (+ 1476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:46.876008 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:46.876665 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:46.876669 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:46.876873 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:46.877668 (+ 795us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:30:46.877672 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:46.877740 (+ 68us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:30:46.877872 (+ 132us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:30:46.878240 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:30:46.878251 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:30:46.878258 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 07:30:46.878390 (+ 132us) client_negotiation.cc:240] Negotiation successful
0926 07:30:46.878742 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":141,"threads_started":1}
I20260926 07:30:46.881362 32319 heartbeater.cc:347] Connected to a master server at 127.22.53.254:40197
I20260926 07:30:46.881675 32319 heartbeater.cc:464] Registering TS with master...
I20260926 07:30:46.882376 32319 heartbeater.cc:511] Master 127.22.53.254:40197 requested a full tablet report, sending...
I20260926 07:30:46.883766 32000 ts_manager.cc:194] Registered new tserver with Master: af48d8b8569b4deea88e325391785af6 (127.22.53.194:34177)
I20260926 07:30:46.884406 32000 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.53.194:59645
W20260926 07:30:47.000854 32322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:30:47.001174 32322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:30:47.001214 32322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:30:47.005630 32322 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:30:47.005704 32322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:30:47.005764 32322 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.53.195
I20260926 07:30:47.011698 32322 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:39697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407783529717-22743-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407783529717-22743-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.53.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407783529717-22743-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.53.195
--webserver_port=0
--tserver_master_addrs=127.22.53.254:40197
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32322
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.53.195
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407783529717-22743-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:30:47.013027 32322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:30:47.014499 32322 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 07:30:47.025218 32322 server_base.cc:1034] running on GCE node
W20260926 07:30:47.025177 32327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:47.025177 32328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:47.025182 32330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:47.025856 32322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:30:47.026590 32322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:30:47.027812 32322 hybrid_clock.cc:648] HybridClock initialized: now 1790407847027776 us; error 49 us; skew 500 ppm
I20260926 07:30:47.030015 32322 webserver.cc:460] Webserver started at http://127.22.53.195:39651/ using document root <none> and password file <none>
I20260926 07:30:47.030682 32322 fs_manager.cc:359] Metadata directory not provided
I20260926 07:30:47.030784 32322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:30:47.031029 32322 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:30:47.033422 32322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407783529717-22743-0/minicluster-data/ts-2/data/instance:
uuid: "7ae2e82a1181452798864032846b78c4"
format_stamp: "Formatted at 2026-09-26 07:30:47 on dist-test-slave-wdjc"
server_key: "7d8976b10aee73599775690a300398cf"
server_key_iv: "90bb70925940e0fe76f8aec03b7779c8"
server_key_version: "encryptionkey@0"
I20260926 07:30:47.034075 32322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407783529717-22743-0/minicluster-data/ts-2/wal/instance:
uuid: "7ae2e82a1181452798864032846b78c4"
format_stamp: "Formatted at 2026-09-26 07:30:47 on dist-test-slave-wdjc"
server_key: "7d8976b10aee73599775690a300398cf"
server_key_iv: "90bb70925940e0fe76f8aec03b7779c8"
server_key_version: "encryptionkey@0"
I20260926 07:30:47.038254 32322 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 07:30:47.041535 32336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:47.043169 32322 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 07:30:47.043352 32322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407783529717-22743-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407783529717-22743-0/minicluster-data/ts-2/wal
uuid: "7ae2e82a1181452798864032846b78c4"
format_stamp: "Formatted at 2026-09-26 07:30:47 on dist-test-slave-wdjc"
server_key: "7d8976b10aee73599775690a300398cf"
server_key_iv: "90bb70925940e0fe76f8aec03b7779c8"
server_key_version: "encryptionkey@0"
I20260926 07:30:47.043509 32322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407783529717-22743-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407783529717-22743-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407783529717-22743-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:30:47.067970 32322 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:30:47.068899 32322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:30:47.069151 32322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:30:47.069836 32322 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:30:47.070967 32322 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:30:47.071063 32322 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:47.071143 32322 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:30:47.071197 32322 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:47.081777 32322 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.195:37759
I20260926 07:30:47.081802 32449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.195:37759 every 8 connection(s)
I20260926 07:30:47.082926 32322 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407783529717-22743-0/minicluster-data/ts-2/data/info.pb
I20260926 07:30:47.086719 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 32322
I20260926 07:30:47.086839 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407783529717-22743-0/minicluster-data/ts-2/wal/instance
I20260926 07:30:47.087242 22743 external_mini_cluster.cc:1428] Setting key 57a35c9b20c45973bd5f43201a29b2e5
I20260926 07:30:47.093814 32039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:47.085127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.195:59929 (local address 127.22.53.254:40197)
0926 07:30:47.085285 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:47.085289 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:47.086435 (+ 1146us) server_negotiation.cc:407] Connection header received
0926 07:30:47.088459 (+ 2024us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:47.088463 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:47.088562 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:47.088655 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:47.089982 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:47.090573 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:47.091796 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:47.092022 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:47.092111 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:47.093360 (+ 1249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:47.093363 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:47.093365 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:30:47.093380 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:47.093399 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:30:47.093407 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:47.093468 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:47.093625 (+ 157us) server_negotiation.cc:299] Negotiation successful
0926 07:30:47.093685 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 07:30:47.094851 32452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:47.085545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:40197 (local address 127.22.53.195:59929)
0926 07:30:47.086248 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:47.086293 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:47.088133 (+ 1840us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:47.088889 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:47.088901 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:47.089027 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:47.089821 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:47.089841 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:47.090698 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:47.090700 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:47.091665 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:47.091676 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:47.092181 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:47.092183 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:47.092347 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:47.093063 (+ 716us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:30:47.093065 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:47.093112 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:30:47.093191 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:30:47.093489 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:30:47.093496 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:30:47.093500 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 07:30:47.093610 (+ 110us) client_negotiation.cc:240] Negotiation successful
0926 07:30:47.093967 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":114,"threads_started":1}
I20260926 07:30:47.096406 32450 heartbeater.cc:347] Connected to a master server at 127.22.53.254:40197
I20260926 07:30:47.096839 32450 heartbeater.cc:464] Registering TS with master...
I20260926 07:30:47.097523 32450 heartbeater.cc:511] Master 127.22.53.254:40197 requested a full tablet report, sending...
I20260926 07:30:47.098809 32000 ts_manager.cc:194] Registered new tserver with Master: 7ae2e82a1181452798864032846b78c4 (127.22.53.195:37759)
I20260926 07:30:47.099447 32000 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.53.195:59929
I20260926 07:30:47.102118 22743 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 07:30:47.105544 32039 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:30:47.104193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38366 (local address 127.22.53.254:40197)
0926 07:30:47.104340 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:47.104344 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:47.104610 (+ 266us) server_negotiation.cc:407] Connection header received
0926 07:30:47.104913 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:47.104916 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:47.104964 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:47.105047 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:47.105419 (+ 372us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38366: BlockingRecv error: recv got EOF from 127.0.0.1:38366 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 07:30:47.105942 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 32060
I20260926 07:30:47.112003 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 32191
I20260926 07:30:47.118211 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 32322
I20260926 07:30:47.124364 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 31969
2026-09-26T07:30:47Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (1051 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-26T07:30:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:30:47Z Disabled control of system clock
I20260926 07:30:47.155808 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407783529717-22743-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407783529717-22743-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407783529717-22743-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.53.254:38565
--webserver_interface=127.22.53.254
--webserver_port=0
--builtin_ntp_servers=127.22.53.212:45563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.53.254:38565
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 07:30:47.276252 32461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:30:47.276691 32461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:30:47.276741 32461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:30:47.281049 32461 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:30:47.281134 32461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:30:47.281149 32461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:30:47.281167 32461 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:30:47.281184 32461 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:30:47.286686 32461 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:45563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407783529717-22743-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407783529717-22743-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.53.254:38565
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.53.254:38565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.53.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32461
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407783529717-22743-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:30:47.287963 32461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:30:47.289556 32461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:30:47.299906 32466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:47.299881 32467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:47.300026 32469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:47.300208 32461 server_base.cc:1034] running on GCE node
I20260926 07:30:47.300994 32461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:30:47.302423 32461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:30:47.303674 32461 hybrid_clock.cc:648] HybridClock initialized: now 1790407847303614 us; error 83 us; skew 500 ppm
I20260926 07:30:47.306358 32461 webserver.cc:460] Webserver started at http://127.22.53.254:41463/ using document root <none> and password file <none>
I20260926 07:30:47.307083 32461 fs_manager.cc:359] Metadata directory not provided
I20260926 07:30:47.307163 32461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:30:47.307413 32461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:30:47.309882 32461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407783529717-22743-0/minicluster-data/master-0/data/instance:
uuid: "70fc25dfa2e646bcad75f6277138d12a"
format_stamp: "Formatted at 2026-09-26 07:30:47 on dist-test-slave-wdjc"
server_key: "0b0c09e61c248e456953b865072c4467"
server_key_iv: "e78a71f2e4162776fdfa8e7bd2ea3b32"
server_key_version: "encryptionkey@0"
I20260926 07:30:47.310571 32461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407783529717-22743-0/minicluster-data/master-0/wal/instance:
uuid: "70fc25dfa2e646bcad75f6277138d12a"
format_stamp: "Formatted at 2026-09-26 07:30:47 on dist-test-slave-wdjc"
server_key: "0b0c09e61c248e456953b865072c4467"
server_key_iv: "e78a71f2e4162776fdfa8e7bd2ea3b32"
server_key_version: "encryptionkey@0"
I20260926 07:30:47.314962 32461 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 07:30:47.317929 32475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:47.319401 32461 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 07:30:47.319581 32461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407783529717-22743-0/minicluster-data/master-0/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407783529717-22743-0/minicluster-data/master-0/wal
uuid: "70fc25dfa2e646bcad75f6277138d12a"
format_stamp: "Formatted at 2026-09-26 07:30:47 on dist-test-slave-wdjc"
server_key: "0b0c09e61c248e456953b865072c4467"
server_key_iv: "e78a71f2e4162776fdfa8e7bd2ea3b32"
server_key_version: "encryptionkey@0"
I20260926 07:30:47.319729 32461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407783529717-22743-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407783529717-22743-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407783529717-22743-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:30:47.325139 32461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:30:47.325877 32461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:30:47.326087 32461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:30:47.334157 32461 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.254:38565
I20260926 07:30:47.334220 32527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.254:38565 every 8 connection(s)
I20260926 07:30:47.335261 32461 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
I20260926 07:30:47.338883 32528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:30:47.343011 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 32461
I20260926 07:30:47.343189 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407783529717-22743-0/minicluster-data/master-0/wal/instance
I20260926 07:30:47.343500 22743 external_mini_cluster.cc:1428] Setting key 212623cc360ea46f4379924f2d066e4d
I20260926 07:30:47.345780 32528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70fc25dfa2e646bcad75f6277138d12a: Bootstrap starting.
I20260926 07:30:47.348775 32528 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 70fc25dfa2e646bcad75f6277138d12a: Neither blocks nor log segments found. Creating new log.
I20260926 07:30:47.349807 32528 log.cc:824] T 00000000000000000000000000000000 P 70fc25dfa2e646bcad75f6277138d12a: Log is configured to *not* fsync() on all Append() calls
I20260926 07:30:47.350548 32531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:47.345224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60818 (local address 127.22.53.254:38565)
0926 07:30:47.346015 (+ 791us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:47.346045 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:47.346093 (+ 48us) server_negotiation.cc:407] Connection header received
0926 07:30:47.347014 (+ 921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:47.347044 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:47.347554 (+ 510us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:47.347958 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:47.348963 (+ 1005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:47.348969 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:47.348985 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:30:47.349000 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:47.349084 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:30:47.349111 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:47.349237 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:47.349373 (+ 136us) server_negotiation.cc:299] Negotiation successful
0926 07:30:47.349638 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":137,"threads_started":1}
I20260926 07:30:47.352357 32528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70fc25dfa2e646bcad75f6277138d12a: No bootstrap required, opened a new log
I20260926 07:30:47.357376 32528 raft_consensus.cc:348] T 00000000000000000000000000000000 P 70fc25dfa2e646bcad75f6277138d12a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70fc25dfa2e646bcad75f6277138d12a" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 38565 } }
I20260926 07:30:47.357728 32528 raft_consensus.cc:374] T 00000000000000000000000000000000 P 70fc25dfa2e646bcad75f6277138d12a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:30:47.357823 32528 raft_consensus.cc:729] T 00000000000000000000000000000000 P 70fc25dfa2e646bcad75f6277138d12a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70fc25dfa2e646bcad75f6277138d12a, State: Initialized, Role: FOLLOWER
I20260926 07:30:47.358340 32528 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70fc25dfa2e646bcad75f6277138d12a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70fc25dfa2e646bcad75f6277138d12a" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 38565 } }
I20260926 07:30:47.358513 32528 raft_consensus.cc:388] T 00000000000000000000000000000000 P 70fc25dfa2e646bcad75f6277138d12a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:30:47.358621 32528 raft_consensus.cc:482] T 00000000000000000000000000000000 P 70fc25dfa2e646bcad75f6277138d12a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:30:47.358763 32528 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 70fc25dfa2e646bcad75f6277138d12a [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:30:47.360221 32528 raft_consensus.cc:504] T 00000000000000000000000000000000 P 70fc25dfa2e646bcad75f6277138d12a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70fc25dfa2e646bcad75f6277138d12a" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 38565 } }
I20260926 07:30:47.360716 32528 leader_election.cc:302] T 00000000000000000000000000000000 P 70fc25dfa2e646bcad75f6277138d12a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 70fc25dfa2e646bcad75f6277138d12a; no voters:
I20260926 07:30:47.361114 32528 leader_election.cc:288] T 00000000000000000000000000000000 P 70fc25dfa2e646bcad75f6277138d12a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:30:47.361370 32533 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 70fc25dfa2e646bcad75f6277138d12a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:30:47.361685 32533 raft_consensus.cc:686] T 00000000000000000000000000000000 P 70fc25dfa2e646bcad75f6277138d12a [term 1 LEADER]: Becoming Leader. State: Replica: 70fc25dfa2e646bcad75f6277138d12a, State: Running, Role: LEADER
I20260926 07:30:47.362056 32533 consensus_queue.cc:237] T 00000000000000000000000000000000 P 70fc25dfa2e646bcad75f6277138d12a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70fc25dfa2e646bcad75f6277138d12a" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 38565 } }
I20260926 07:30:47.362661 32528 sys_catalog.cc:564] T 00000000000000000000000000000000 P 70fc25dfa2e646bcad75f6277138d12a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:30:47.363914 32535 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70fc25dfa2e646bcad75f6277138d12a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70fc25dfa2e646bcad75f6277138d12a. Latest consensus state: current_term: 1 leader_uuid: "70fc25dfa2e646bcad75f6277138d12a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70fc25dfa2e646bcad75f6277138d12a" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 38565 } } }
I20260926 07:30:47.363978 32534 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70fc25dfa2e646bcad75f6277138d12a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "70fc25dfa2e646bcad75f6277138d12a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70fc25dfa2e646bcad75f6277138d12a" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 38565 } } }
I20260926 07:30:47.364061 32535 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70fc25dfa2e646bcad75f6277138d12a [sys.catalog]: This master's current role is: LEADER
I20260926 07:30:47.364089 32534 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70fc25dfa2e646bcad75f6277138d12a [sys.catalog]: This master's current role is: LEADER
I20260926 07:30:47.364490 32542 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:30:47.368432 32542 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:30:47.375993 32542 catalog_manager.cc:1347] Generated new cluster ID: 5dc2ef70492a4347ad91aeae1b7b7de1
I20260926 07:30:47.376106 32542 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:30:47.389308 32542 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:30:47.389727 32542 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:30:47.399473 32542 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 70fc25dfa2e646bcad75f6277138d12a: Generated new TSK 0
I20260926 07:30:47.400398 32542 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:30:47.411760 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407783529717-22743-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407783529717-22743-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407783529717-22743-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407783529717-22743-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.53.193:0
--local_ip_for_outbound_sockets=127.22.53.193
--webserver_interface=127.22.53.193
--webserver_port=0
--tserver_master_addrs=127.22.53.254:38565
--builtin_ntp_servers=127.22.53.212:45563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 07:30:47.547747 32552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:30:47.548070 32552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:30:47.548115 32552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:30:47.552564 32552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:30:47.552639 32552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:30:47.552731 32552 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.53.193
I20260926 07:30:47.557711 32552 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:45563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407783529717-22743-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407783529717-22743-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.53.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407783529717-22743-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.53.193
--webserver_port=0
--tserver_master_addrs=127.22.53.254:38565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32552
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.53.193
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407783529717-22743-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:30:47.558818 32552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:30:47.560297 32552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:30:47.571744 32558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:47.571846 32552 server_base.cc:1034] running on GCE node
W20260926 07:30:47.571770 32557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:47.571784 32560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:47.572635 32552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:30:47.573380 32552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:30:47.574671 32552 hybrid_clock.cc:648] HybridClock initialized: now 1790407847574662 us; error 57 us; skew 500 ppm
I20260926 07:30:47.577075 32552 webserver.cc:460] Webserver started at http://127.22.53.193:41969/ using document root <none> and password file <none>
I20260926 07:30:47.577786 32552 fs_manager.cc:359] Metadata directory not provided
I20260926 07:30:47.577870 32552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:30:47.578130 32552 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:30:47.580816 32552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407783529717-22743-0/minicluster-data/ts-0/data/instance:
uuid: "8affb3bcc8b24c74a415f0809359fc2f"
format_stamp: "Formatted at 2026-09-26 07:30:47 on dist-test-slave-wdjc"
server_key: "0e966f5a8d4c2f936f2f1eef6eabd480"
server_key_iv: "e9ece52cc245467649b5504eb66f8466"
server_key_version: "encryptionkey@0"
I20260926 07:30:47.581563 32552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407783529717-22743-0/minicluster-data/ts-0/wal/instance:
uuid: "8affb3bcc8b24c74a415f0809359fc2f"
format_stamp: "Formatted at 2026-09-26 07:30:47 on dist-test-slave-wdjc"
server_key: "0e966f5a8d4c2f936f2f1eef6eabd480"
server_key_iv: "e9ece52cc245467649b5504eb66f8466"
server_key_version: "encryptionkey@0"
I20260926 07:30:47.587234 32552 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 07:30:47.590617 32566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:47.592253 32552 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 07:30:47.592435 32552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407783529717-22743-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407783529717-22743-0/minicluster-data/ts-0/wal
uuid: "8affb3bcc8b24c74a415f0809359fc2f"
format_stamp: "Formatted at 2026-09-26 07:30:47 on dist-test-slave-wdjc"
server_key: "0e966f5a8d4c2f936f2f1eef6eabd480"
server_key_iv: "e9ece52cc245467649b5504eb66f8466"
server_key_version: "encryptionkey@0"
I20260926 07:30:47.592665 32552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407783529717-22743-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407783529717-22743-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407783529717-22743-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:30:47.598032 32552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:30:47.598816 32552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:30:47.599048 32552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:30:47.599696 32552 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:30:47.600966 32552 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:30:47.601045 32552 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:47.601125 32552 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:30:47.601178 32552 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:47.612030 32552 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.193:36803
I20260926 07:30:47.612040 32679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.193:36803 every 8 connection(s)
I20260926 07:30:47.613191 32552 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407783529717-22743-0/minicluster-data/ts-0/data/info.pb
I20260926 07:30:47.620434 32531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:47.615344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.193:50079 (local address 127.22.53.254:38565)
0926 07:30:47.615535 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:47.615539 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:47.616841 (+ 1302us) server_negotiation.cc:407] Connection header received
0926 07:30:47.618400 (+ 1559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:47.618403 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:47.618458 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:47.618559 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:47.619889 (+ 1330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:47.619893 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:47.619896 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:30:47.619918 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:47.619941 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:30:47.619950 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:47.620068 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:47.620223 (+ 155us) server_negotiation.cc:299] Negotiation successful
0926 07:30:47.620280 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 07:30:47.620919 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 32552
I20260926 07:30:47.621057 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407783529717-22743-0/minicluster-data/ts-0/wal/instance
I20260926 07:30:47.621328 22743 external_mini_cluster.cc:1428] Setting key 24bc4570a76605b9450534c54481feaa
I20260926 07:30:47.622464 32682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:47.615759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:38565 (local address 127.22.53.193:50079)
0926 07:30:47.616663 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:47.616715 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:47.618125 (+ 1410us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:47.618773 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:47.618781 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:47.618885 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:47.619559 (+ 674us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:30:47.619561 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:47.619607 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:30:47.619689 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:30:47.620045 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:30:47.620054 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:30:47.620060 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 07:30:47.620276 (+ 216us) client_negotiation.cc:240] Negotiation successful
0926 07:30:47.620610 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":121,"threads_started":1}
I20260926 07:30:47.623570 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407783529717-22743-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407783529717-22743-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407783529717-22743-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407783529717-22743-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.53.194:0
--local_ip_for_outbound_sockets=127.22.53.194
--webserver_interface=127.22.53.194
--webserver_port=0
--tserver_master_addrs=127.22.53.254:38565
--builtin_ntp_servers=127.22.53.212:45563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260926 07:30:47.624051 32680 heartbeater.cc:347] Connected to a master server at 127.22.53.254:38565
I20260926 07:30:47.624329 32680 heartbeater.cc:464] Registering TS with master...
I20260926 07:30:47.624866 32680 heartbeater.cc:511] Master 127.22.53.254:38565 requested a full tablet report, sending...
I20260926 07:30:47.626852 32492 ts_manager.cc:194] Registered new tserver with Master: 8affb3bcc8b24c74a415f0809359fc2f (127.22.53.193:36803)
W20260926 07:30:47.747575 32683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:30:47.747900 32683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:30:47.747943 32683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:30:47.752521 32683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:30:47.752641 32683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:30:47.752703 32683 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.53.194
I20260926 07:30:47.758220 32683 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:45563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407783529717-22743-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407783529717-22743-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.53.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407783529717-22743-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.53.194
--webserver_port=0
--tserver_master_addrs=127.22.53.254:38565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32683
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.53.194
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407783529717-22743-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:30:47.759507 32683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:30:47.761111 32683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:30:47.772475 32688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:47.772485 32689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:47.772554 32683 server_base.cc:1034] running on GCE node
W20260926 07:30:47.772485 32691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:47.773249 32683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:30:47.774080 32683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:30:47.775281 32683 hybrid_clock.cc:648] HybridClock initialized: now 1790407847775264 us; error 33 us; skew 500 ppm
I20260926 07:30:47.777787 32683 webserver.cc:460] Webserver started at http://127.22.53.194:34489/ using document root <none> and password file <none>
I20260926 07:30:47.778471 32683 fs_manager.cc:359] Metadata directory not provided
I20260926 07:30:47.778581 32683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:30:47.778854 32683 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:30:47.781276 32683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407783529717-22743-0/minicluster-data/ts-1/data/instance:
uuid: "b87b01fd0fb242aab53399be302fc16d"
format_stamp: "Formatted at 2026-09-26 07:30:47 on dist-test-slave-wdjc"
server_key: "9c9edb829be356388a29d911bde6cae5"
server_key_iv: "9ea410d1281680ab5777bb35441383d0"
server_key_version: "encryptionkey@0"
I20260926 07:30:47.781956 32683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407783529717-22743-0/minicluster-data/ts-1/wal/instance:
uuid: "b87b01fd0fb242aab53399be302fc16d"
format_stamp: "Formatted at 2026-09-26 07:30:47 on dist-test-slave-wdjc"
server_key: "9c9edb829be356388a29d911bde6cae5"
server_key_iv: "9ea410d1281680ab5777bb35441383d0"
server_key_version: "encryptionkey@0"
I20260926 07:30:47.787005 32683 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 07:30:47.790619 32697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:47.792449 32683 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 07:30:47.792681 32683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407783529717-22743-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407783529717-22743-0/minicluster-data/ts-1/wal
uuid: "b87b01fd0fb242aab53399be302fc16d"
format_stamp: "Formatted at 2026-09-26 07:30:47 on dist-test-slave-wdjc"
server_key: "9c9edb829be356388a29d911bde6cae5"
server_key_iv: "9ea410d1281680ab5777bb35441383d0"
server_key_version: "encryptionkey@0"
I20260926 07:30:47.792850 32683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407783529717-22743-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407783529717-22743-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407783529717-22743-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:30:47.798539 32683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:30:47.799324 32683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:30:47.799554 32683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:30:47.800223 32683 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:30:47.801421 32683 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:30:47.801498 32683 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:47.801578 32683 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:30:47.801621 32683 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:47.811677 32683 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.194:36895
I20260926 07:30:47.811707 342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.194:36895 every 8 connection(s)
I20260926 07:30:47.812760 32683 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407783529717-22743-0/minicluster-data/ts-1/data/info.pb
I20260926 07:30:47.819952 32531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:47.814904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.194:46887 (local address 127.22.53.254:38565)
0926 07:30:47.815071 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:47.815075 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:47.816252 (+ 1177us) server_negotiation.cc:407] Connection header received
0926 07:30:47.817754 (+ 1502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:47.817757 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:47.817804 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:47.817912 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:47.819323 (+ 1411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:47.819326 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:47.819329 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:30:47.819363 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:47.819388 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:30:47.819397 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:47.819501 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:47.819777 (+ 276us) server_negotiation.cc:299] Negotiation successful
0926 07:30:47.819827 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 07:30:47.821111 345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:47.815325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:38565 (local address 127.22.53.194:46887)
0926 07:30:47.816074 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:47.816120 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:47.817471 (+ 1351us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:47.818159 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:47.818168 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:47.818276 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:47.818993 (+ 717us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:30:47.818995 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:47.819050 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:30:47.819133 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:30:47.819554 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:30:47.819584 (+ 30us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:30:47.819590 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 07:30:47.819787 (+ 197us) client_negotiation.cc:240] Negotiation successful
0926 07:30:47.820062 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":119,"threads_started":1}
I20260926 07:30:47.821605 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 32683
I20260926 07:30:47.821749 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407783529717-22743-0/minicluster-data/ts-1/wal/instance
I20260926 07:30:47.822093 22743 external_mini_cluster.cc:1428] Setting key b6b4f1a8b1c97c12a003f33b97cce0cf
I20260926 07:30:47.822633 343 heartbeater.cc:347] Connected to a master server at 127.22.53.254:38565
I20260926 07:30:47.822995 343 heartbeater.cc:464] Registering TS with master...
I20260926 07:30:47.823565 343 heartbeater.cc:511] Master 127.22.53.254:38565 requested a full tablet report, sending...
I20260926 07:30:47.824616 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407783529717-22743-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407783529717-22743-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407783529717-22743-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407783529717-22743-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.53.195:0
--local_ip_for_outbound_sockets=127.22.53.195
--webserver_interface=127.22.53.195
--webserver_port=0
--tserver_master_addrs=127.22.53.254:38565
--builtin_ntp_servers=127.22.53.212:45563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260926 07:30:47.824966 32492 ts_manager.cc:194] Registered new tserver with Master: b87b01fd0fb242aab53399be302fc16d (127.22.53.194:36895)
W20260926 07:30:47.944717 346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:30:47.945042 346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:30:47.945129 346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:30:47.949400 346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:30:47.949501 346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:30:47.949610 346 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.53.195
I20260926 07:30:47.954718 346 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:45563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407783529717-22743-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407783529717-22743-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.53.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407783529717-22743-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.53.195
--webserver_port=0
--tserver_master_addrs=127.22.53.254:38565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.346
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.53.195
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407783529717-22743-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:30:47.955951 346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:30:47.957535 346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:30:47.968713 352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:47.968706 354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:47.968706 351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:47.969139 346 server_base.cc:1034] running on GCE node
I20260926 07:30:47.969692 346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:30:47.970523 346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:30:47.971782 346 hybrid_clock.cc:648] HybridClock initialized: now 1790407847971739 us; error 57 us; skew 500 ppm
I20260926 07:30:47.978448 346 webserver.cc:460] Webserver started at http://127.22.53.195:37199/ using document root <none> and password file <none>
I20260926 07:30:47.979153 346 fs_manager.cc:359] Metadata directory not provided
I20260926 07:30:47.979266 346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:30:47.979671 346 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:30:47.982149 346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407783529717-22743-0/minicluster-data/ts-2/data/instance:
uuid: "8b7a214cc14046cd8ada7f1067faa8a6"
format_stamp: "Formatted at 2026-09-26 07:30:47 on dist-test-slave-wdjc"
server_key: "a796ec58176acbc3f43b23e65da788f6"
server_key_iv: "73eaea628a9431f9b06d6f8c3d02ab4a"
server_key_version: "encryptionkey@0"
I20260926 07:30:47.982780 346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407783529717-22743-0/minicluster-data/ts-2/wal/instance:
uuid: "8b7a214cc14046cd8ada7f1067faa8a6"
format_stamp: "Formatted at 2026-09-26 07:30:47 on dist-test-slave-wdjc"
server_key: "a796ec58176acbc3f43b23e65da788f6"
server_key_iv: "73eaea628a9431f9b06d6f8c3d02ab4a"
server_key_version: "encryptionkey@0"
I20260926 07:30:47.986948 346 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 07:30:47.989933 360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:47.991446 346 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 07:30:47.991629 346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407783529717-22743-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407783529717-22743-0/minicluster-data/ts-2/wal
uuid: "8b7a214cc14046cd8ada7f1067faa8a6"
format_stamp: "Formatted at 2026-09-26 07:30:47 on dist-test-slave-wdjc"
server_key: "a796ec58176acbc3f43b23e65da788f6"
server_key_iv: "73eaea628a9431f9b06d6f8c3d02ab4a"
server_key_version: "encryptionkey@0"
I20260926 07:30:47.991778 346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407783529717-22743-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407783529717-22743-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407783529717-22743-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:30:47.997246 346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:30:47.998116 346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:30:47.998333 346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:30:47.998968 346 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:30:47.999997 346 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:30:48.000077 346 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:48.000154 346 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:30:48.000200 346 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:48.011592 346 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.195:33141
I20260926 07:30:48.011761 473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.195:33141 every 8 connection(s)
I20260926 07:30:48.012862 346 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407783529717-22743-0/minicluster-data/ts-2/data/info.pb
I20260926 07:30:48.020744 32531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:48.015160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.195:43563 (local address 127.22.53.254:38565)
0926 07:30:48.015354 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:48.015357 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:48.016791 (+ 1434us) server_negotiation.cc:407] Connection header received
0926 07:30:48.018323 (+ 1532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:48.018328 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:48.018413 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:48.018553 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:48.020101 (+ 1548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:48.020105 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:48.020107 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:30:48.020128 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:48.020161 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:30:48.020172 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:48.020276 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:48.020466 (+ 190us) server_negotiation.cc:299] Negotiation successful
0926 07:30:48.020603 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260926 07:30:48.021749 476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:48.015667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:38565 (local address 127.22.53.195:43563)
0926 07:30:48.016514 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:48.016653 (+ 139us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:48.017956 (+ 1303us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:48.018817 (+ 861us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:48.018827 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:48.018936 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:48.019693 (+ 757us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:30:48.019694 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:48.019742 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:30:48.019828 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:30:48.020281 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:30:48.020290 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:30:48.020295 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 07:30:48.020525 (+ 230us) client_negotiation.cc:240] Negotiation successful
0926 07:30:48.020845 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":155,"threads_started":1}
I20260926 07:30:48.022253 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 346
I20260926 07:30:48.022382 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407783529717-22743-0/minicluster-data/ts-2/wal/instance
I20260926 07:30:48.022730 22743 external_mini_cluster.cc:1428] Setting key 8dbcc6723d40e1e9de1109cc778da2dc
I20260926 07:30:48.023367 474 heartbeater.cc:347] Connected to a master server at 127.22.53.254:38565
I20260926 07:30:48.023656 474 heartbeater.cc:464] Registering TS with master...
I20260926 07:30:48.024185 474 heartbeater.cc:511] Master 127.22.53.254:38565 requested a full tablet report, sending...
I20260926 07:30:48.025489 32492 ts_manager.cc:194] Registered new tserver with Master: 8b7a214cc14046cd8ada7f1067faa8a6 (127.22.53.195:33141)
I20260926 07:30:48.037778 22743 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 07:30:48.041626 32531 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:30:48.040110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60834 (local address 127.22.53.254:38565)
0926 07:30:48.040279 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:48.040283 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:48.040416 (+ 133us) server_negotiation.cc:407] Connection header received
0926 07:30:48.040593 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:48.040596 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:48.040637 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:48.040720 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:48.041459 (+ 739us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60834: BlockingRecv error: recv got EOF from 127.0.0.1:60834 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 07:30:48.041615 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 32552
I20260926 07:30:48.047775 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 32683
I20260926 07:30:48.053855 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 346
I20260926 07:30:48.059741 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 32461
2026-09-26T07:30:48Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (935 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407783529717-22743-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:30:48 dist-test-slave-wdjc krb5kdc[484](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:30:48 dist-test-slave-wdjc krb5kdc[484](info): set up 2 sockets
Sep 26 07:30:48 dist-test-slave-wdjc krb5kdc[484](info): commencing operation
krb5kdc: starting...
W20260926 07:30:49.117815 22743 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407783529717-22743-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407783529717-22743-0/krb5kdc/test-user.keytab.
Sep 26 07:30:49 dist-test-slave-wdjc krb5kdc[484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407849, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:30:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:30:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.53.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.53.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.53.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab.
Entry for principal kudu/127.22.53.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab.
I20260926 07:30:49.248272 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407783529717-22743-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407783529717-22743-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407783529717-22743-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.53.254:46439
--webserver_interface=127.22.53.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab
--principal=kudu/127.22.53.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:42697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.53.254:46439
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:30:49.365859 497 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:30:49.366228 497 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:30:49.366310 497 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:30:49.370626 497 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:30:49.370762 497 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:30:49.370798 497 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:30:49.370837 497 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:30:49.370893 497 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:30:49.376616 497 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:42697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407783529717-22743-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407783529717-22743-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.53.254:46439
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab
--principal=kudu/127.22.53.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.53.254:46439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.53.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.497
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407783529717-22743-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:30:49.377849 497 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:30:49.378943 497 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:30:49.388152 505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:49.388288 497 server_base.cc:1034] running on GCE node
W20260926 07:30:49.388077 503 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:49.388190 502 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:49.389062 497 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:30:49.390298 497 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:30:49.391491 497 hybrid_clock.cc:648] HybridClock initialized: now 1790407849391479 us; error 31 us; skew 500 ppm
Sep 26 07:30:49 dist-test-slave-wdjc krb5kdc[484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407849, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:30:49.394662 497 init.cc:377] Logged in from keytab as kudu/127.22.53.254@KRBTEST.COM (short username kudu)
I20260926 07:30:49.395701 497 webserver.cc:460] Webserver started at http://127.22.53.254:43657/ using document root <none> and password file <none>
I20260926 07:30:49.396377 497 fs_manager.cc:359] Metadata directory not provided
I20260926 07:30:49.396462 497 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:30:49.396715 497 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:30:49.398862 497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407783529717-22743-0/minicluster-data/master-0/data/instance:
uuid: "500d83d94ada4846b152ca2d5f9f2f51"
format_stamp: "Formatted at 2026-09-26 07:30:49 on dist-test-slave-wdjc"
server_key: "84d9787a1c34b98eea40fa6ee25449e9"
server_key_iv: "fd4dc9d775577395c8331a69b92b2875"
server_key_version: "encryptionkey@0"
I20260926 07:30:49.399420 497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407783529717-22743-0/minicluster-data/master-0/wal/instance:
uuid: "500d83d94ada4846b152ca2d5f9f2f51"
format_stamp: "Formatted at 2026-09-26 07:30:49 on dist-test-slave-wdjc"
server_key: "84d9787a1c34b98eea40fa6ee25449e9"
server_key_iv: "fd4dc9d775577395c8331a69b92b2875"
server_key_version: "encryptionkey@0"
I20260926 07:30:49.403625 497 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 07:30:49.406517 512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:49.408106 497 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 07:30:49.408288 497 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407783529717-22743-0/minicluster-data/master-0/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407783529717-22743-0/minicluster-data/master-0/wal
uuid: "500d83d94ada4846b152ca2d5f9f2f51"
format_stamp: "Formatted at 2026-09-26 07:30:49 on dist-test-slave-wdjc"
server_key: "84d9787a1c34b98eea40fa6ee25449e9"
server_key_iv: "fd4dc9d775577395c8331a69b92b2875"
server_key_version: "encryptionkey@0"
I20260926 07:30:49.408438 497 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407783529717-22743-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407783529717-22743-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407783529717-22743-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:30:49.419566 497 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:30:49.423038 497 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:30:49.423278 497 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:30:49.431427 497 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.254:46439
I20260926 07:30:49.431473 564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.254:46439 every 8 connection(s)
I20260926 07:30:49.432583 497 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
I20260926 07:30:49.435768 565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:30:49.435865 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 497
I20260926 07:30:49.435973 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407783529717-22743-0/minicluster-data/master-0/wal/instance
I20260926 07:30:49.436277 22743 external_mini_cluster.cc:1428] Setting key aef35250361e93a4c06ad044c87e63c3
I20260926 07:30:49.443038 565 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 500d83d94ada4846b152ca2d5f9f2f51: Bootstrap starting.
Sep 26 07:30:49 dist-test-slave-wdjc krb5kdc[484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407849, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.53.254@KRBTEST.COM
I20260926 07:30:49.445978 565 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 500d83d94ada4846b152ca2d5f9f2f51: Neither blocks nor log segments found. Creating new log.
I20260926 07:30:49.446831 565 log.cc:824] T 00000000000000000000000000000000 P 500d83d94ada4846b152ca2d5f9f2f51: Log is configured to *not* fsync() on all Append() calls
I20260926 07:30:49.449186 565 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 500d83d94ada4846b152ca2d5f9f2f51: No bootstrap required, opened a new log
I20260926 07:30:49.451829 568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:49.437785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53786 (local address 127.22.53.254:46439)
0926 07:30:49.438229 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:49.438245 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:49.438284 (+ 39us) server_negotiation.cc:407] Connection header received
0926 07:30:49.439125 (+ 841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:49.439143 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:49.439492 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:49.439775 (+ 283us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:49.440735 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:49.441777 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:49.442962 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:49.443556 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:49.443684 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:49.446640 (+ 2956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:49.446645 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:49.446661 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:49.446693 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:49.448690 (+ 1997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:49.449199 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:49.449204 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:49.449209 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:49.449442 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:49.449753 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:49.449757 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:49.449759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:49.450206 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:49.450418 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:49.450731 (+ 313us) server_negotiation.cc:299] Negotiation successful
0926 07:30:49.451066 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":145,"threads_started":1}
I20260926 07:30:49.453251 565 raft_consensus.cc:348] T 00000000000000000000000000000000 P 500d83d94ada4846b152ca2d5f9f2f51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "500d83d94ada4846b152ca2d5f9f2f51" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 46439 } }
I20260926 07:30:49.453564 565 raft_consensus.cc:374] T 00000000000000000000000000000000 P 500d83d94ada4846b152ca2d5f9f2f51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:30:49.453616 565 raft_consensus.cc:729] T 00000000000000000000000000000000 P 500d83d94ada4846b152ca2d5f9f2f51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 500d83d94ada4846b152ca2d5f9f2f51, State: Initialized, Role: FOLLOWER
I20260926 07:30:49.454077 565 consensus_queue.cc:260] T 00000000000000000000000000000000 P 500d83d94ada4846b152ca2d5f9f2f51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "500d83d94ada4846b152ca2d5f9f2f51" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 46439 } }
I20260926 07:30:49.454272 565 raft_consensus.cc:388] T 00000000000000000000000000000000 P 500d83d94ada4846b152ca2d5f9f2f51 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:30:49.454340 565 raft_consensus.cc:482] T 00000000000000000000000000000000 P 500d83d94ada4846b152ca2d5f9f2f51 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:30:49.454430 565 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 500d83d94ada4846b152ca2d5f9f2f51 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:30:49.455430 565 raft_consensus.cc:504] T 00000000000000000000000000000000 P 500d83d94ada4846b152ca2d5f9f2f51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "500d83d94ada4846b152ca2d5f9f2f51" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 46439 } }
I20260926 07:30:49.455755 565 leader_election.cc:302] T 00000000000000000000000000000000 P 500d83d94ada4846b152ca2d5f9f2f51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 500d83d94ada4846b152ca2d5f9f2f51; no voters:
I20260926 07:30:49.456032 565 leader_election.cc:288] T 00000000000000000000000000000000 P 500d83d94ada4846b152ca2d5f9f2f51 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:30:49.456231 570 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 500d83d94ada4846b152ca2d5f9f2f51 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:30:49.456529 570 raft_consensus.cc:686] T 00000000000000000000000000000000 P 500d83d94ada4846b152ca2d5f9f2f51 [term 1 LEADER]: Becoming Leader. State: Replica: 500d83d94ada4846b152ca2d5f9f2f51, State: Running, Role: LEADER
I20260926 07:30:49.456919 570 consensus_queue.cc:237] T 00000000000000000000000000000000 P 500d83d94ada4846b152ca2d5f9f2f51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "500d83d94ada4846b152ca2d5f9f2f51" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 46439 } }
I20260926 07:30:49.457435 565 sys_catalog.cc:564] T 00000000000000000000000000000000 P 500d83d94ada4846b152ca2d5f9f2f51 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:30:49.458534 572 sys_catalog.cc:455] T 00000000000000000000000000000000 P 500d83d94ada4846b152ca2d5f9f2f51 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 500d83d94ada4846b152ca2d5f9f2f51. Latest consensus state: current_term: 1 leader_uuid: "500d83d94ada4846b152ca2d5f9f2f51" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "500d83d94ada4846b152ca2d5f9f2f51" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 46439 } } }
I20260926 07:30:49.458665 572 sys_catalog.cc:458] T 00000000000000000000000000000000 P 500d83d94ada4846b152ca2d5f9f2f51 [sys.catalog]: This master's current role is: LEADER
I20260926 07:30:49.459021 571 sys_catalog.cc:455] T 00000000000000000000000000000000 P 500d83d94ada4846b152ca2d5f9f2f51 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "500d83d94ada4846b152ca2d5f9f2f51" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "500d83d94ada4846b152ca2d5f9f2f51" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 46439 } } }
I20260926 07:30:49.459117 571 sys_catalog.cc:458] T 00000000000000000000000000000000 P 500d83d94ada4846b152ca2d5f9f2f51 [sys.catalog]: This master's current role is: LEADER
I20260926 07:30:49.459365 579 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:30:49.462723 579 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:30:49.469615 579 catalog_manager.cc:1347] Generated new cluster ID: ae9ccd821d7248e483400e0866adacae
I20260926 07:30:49.469709 579 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:30:49.481709 579 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:30:49.482765 579 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:30:49.494302 579 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 500d83d94ada4846b152ca2d5f9f2f51: Generated new TSK 0
I20260926 07:30:49.495255 579 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.53.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.53.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.53.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.193.keytab.
Entry for principal kudu/127.22.53.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.193.keytab.
I20260926 07:30:49.536116 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407783529717-22743-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407783529717-22743-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407783529717-22743-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407783529717-22743-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.53.193:0
--local_ip_for_outbound_sockets=127.22.53.193
--webserver_interface=127.22.53.193
--webserver_port=0
--tserver_master_addrs=127.22.53.254:46439
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.193.keytab
--principal=kudu/127.22.53.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:42697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:30:49.657238 591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:30:49.657625 591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:30:49.657675 591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:30:49.662209 591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:30:49.662293 591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:30:49.662362 591 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.53.193
I20260926 07:30:49.668555 591 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:42697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407783529717-22743-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407783529717-22743-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.193.keytab
--principal=kudu/127.22.53.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.53.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407783529717-22743-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.53.193
--webserver_port=0
--tserver_master_addrs=127.22.53.254:46439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.591
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.53.193
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407783529717-22743-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:30:49.669770 591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:30:49.670892 591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:30:49.681552 596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:49.681552 599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:49.681546 597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:49.681958 591 server_base.cc:1034] running on GCE node
I20260926 07:30:49.682472 591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:30:49.683172 591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:30:49.684378 591 hybrid_clock.cc:648] HybridClock initialized: now 1790407849684336 us; error 60 us; skew 500 ppm
Sep 26 07:30:49 dist-test-slave-wdjc krb5kdc[484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407849, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:30:49.687985 591 init.cc:377] Logged in from keytab as kudu/127.22.53.193@KRBTEST.COM (short username kudu)
I20260926 07:30:49.689113 591 webserver.cc:460] Webserver started at http://127.22.53.193:35877/ using document root <none> and password file <none>
I20260926 07:30:49.689754 591 fs_manager.cc:359] Metadata directory not provided
I20260926 07:30:49.689855 591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:30:49.690099 591 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:30:49.692343 591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407783529717-22743-0/minicluster-data/ts-0/data/instance:
uuid: "3eeed72654284329a7b14dc24092d366"
format_stamp: "Formatted at 2026-09-26 07:30:49 on dist-test-slave-wdjc"
server_key: "58e53bd54f6651f288dfc461fb18427a"
server_key_iv: "86a00440619d7ebdeeefa07fc35ad26e"
server_key_version: "encryptionkey@0"
I20260926 07:30:49.693056 591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407783529717-22743-0/minicluster-data/ts-0/wal/instance:
uuid: "3eeed72654284329a7b14dc24092d366"
format_stamp: "Formatted at 2026-09-26 07:30:49 on dist-test-slave-wdjc"
server_key: "58e53bd54f6651f288dfc461fb18427a"
server_key_iv: "86a00440619d7ebdeeefa07fc35ad26e"
server_key_version: "encryptionkey@0"
I20260926 07:30:49.697238 591 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 07:30:49.699971 606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:49.701413 591 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:30:49.701594 591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407783529717-22743-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407783529717-22743-0/minicluster-data/ts-0/wal
uuid: "3eeed72654284329a7b14dc24092d366"
format_stamp: "Formatted at 2026-09-26 07:30:49 on dist-test-slave-wdjc"
server_key: "58e53bd54f6651f288dfc461fb18427a"
server_key_iv: "86a00440619d7ebdeeefa07fc35ad26e"
server_key_version: "encryptionkey@0"
I20260926 07:30:49.701741 591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407783529717-22743-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407783529717-22743-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407783529717-22743-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:30:49.719779 591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:30:49.723186 591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:30:49.723412 591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:30:49.724078 591 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:30:49.725036 591 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:30:49.725109 591 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:49.725180 591 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:30:49.725221 591 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:49.734766 591 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.193:42315
I20260926 07:30:49.734797 719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.193:42315 every 8 connection(s)
I20260926 07:30:49.735703 591 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407783529717-22743-0/minicluster-data/ts-0/data/info.pb
I20260926 07:30:49.743676 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 591
I20260926 07:30:49.743826 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407783529717-22743-0/minicluster-data/ts-0/wal/instance
I20260926 07:30:49.744109 22743 external_mini_cluster.cc:1428] Setting key 72cf11ff654c7bd8a2f5ee4bd1326850
Sep 26 07:30:49 dist-test-slave-wdjc krb5kdc[484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407849, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.193@KRBTEST.COM for kudu/127.22.53.254@KRBTEST.COM
I20260926 07:30:49.751449 568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:49.737798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.193:51543 (local address 127.22.53.254:46439)
0926 07:30:49.737949 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:49.737952 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:49.738840 (+ 888us) server_negotiation.cc:407] Connection header received
0926 07:30:49.740303 (+ 1463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:49.740307 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:49.740372 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:49.740484 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:49.741920 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:49.742479 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:49.743847 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:49.744129 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:49.744257 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:49.747183 (+ 2926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:49.747187 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:49.747189 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:49.747215 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:49.748940 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:49.749552 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:49.749560 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:49.749564 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:49.749664 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:49.750044 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:49.750049 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:49.750054 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:49.750345 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:49.750458 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:49.751015 (+ 557us) server_negotiation.cc:299] Negotiation successful
0926 07:30:49.751188 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 07:30:49.752197 722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:49.738153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:46439 (local address 127.22.53.193:51543)
0926 07:30:49.738667 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:49.738714 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:49.740069 (+ 1355us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:49.740708 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:49.740715 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:49.741117 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:49.741748 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:49.741766 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:49.742625 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:49.742633 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:49.743707 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:49.743718 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:49.744313 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:49.744317 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:49.744579 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:49.745306 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:49.745324 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:49.746986 (+ 1662us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:30:49.749109 (+ 2123us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:49.749115 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:49.749131 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:49.749439 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:49.749762 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:49.749765 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:49.749766 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:49.749906 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:49.750455 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:30:49.750461 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:30:49.750732 (+ 271us) client_negotiation.cc:769] Sending connection context
0926 07:30:49.750905 (+ 173us) client_negotiation.cc:240] Negotiation successful
0926 07:30:49.751209 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":162,"threads_started":1}
I20260926 07:30:49.753688 720 heartbeater.cc:347] Connected to a master server at 127.22.53.254:46439
I20260926 07:30:49.754024 720 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.22.53.194@KRBTEST.COM; defaulting to no policy
I20260926 07:30:49.754770 720 heartbeater.cc:511] Master 127.22.53.254:46439 requested a full tablet report, sending...
I20260926 07:30:49.756817 529 ts_manager.cc:194] Registered new tserver with Master: 3eeed72654284329a7b14dc24092d366 (127.22.53.193:42315)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.53.194@KRBTEST.COM" created.
I20260926 07:30:49.758607 529 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.53.193@KRBTEST.COM'} at 127.22.53.193:51543
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.53.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.194.keytab.
Entry for principal kudu/127.22.53.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.194.keytab.
I20260926 07:30:49.776211 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407783529717-22743-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407783529717-22743-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407783529717-22743-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407783529717-22743-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.53.194:0
--local_ip_for_outbound_sockets=127.22.53.194
--webserver_interface=127.22.53.194
--webserver_port=0
--tserver_master_addrs=127.22.53.254:46439
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.194.keytab
--principal=kudu/127.22.53.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:42697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:30:49.891064 725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:30:49.891428 725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:30:49.891511 725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:30:49.895815 725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:30:49.895931 725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:30:49.896034 725 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.53.194
I20260926 07:30:49.901840 725 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:42697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407783529717-22743-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407783529717-22743-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.194.keytab
--principal=kudu/127.22.53.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.53.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407783529717-22743-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.53.194
--webserver_port=0
--tserver_master_addrs=127.22.53.254:46439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.725
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.53.194
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407783529717-22743-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:30:49.903034 725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:30:49.904125 725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:30:49.915439 730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:49.915673 725 server_base.cc:1034] running on GCE node
W20260926 07:30:49.915445 733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:49.915442 731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:49.916249 725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:30:49.917021 725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:30:49.918246 725 hybrid_clock.cc:648] HybridClock initialized: now 1790407849918195 us; error 55 us; skew 500 ppm
Sep 26 07:30:49 dist-test-slave-wdjc krb5kdc[484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407849, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:30:49.921595 725 init.cc:377] Logged in from keytab as kudu/127.22.53.194@KRBTEST.COM (short username kudu)
I20260926 07:30:49.922644 725 webserver.cc:460] Webserver started at http://127.22.53.194:39959/ using document root <none> and password file <none>
I20260926 07:30:49.923260 725 fs_manager.cc:359] Metadata directory not provided
I20260926 07:30:49.923358 725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:30:49.923636 725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:30:49.926108 725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407783529717-22743-0/minicluster-data/ts-1/data/instance:
uuid: "225f68f5a3334808b45a1756b3734718"
format_stamp: "Formatted at 2026-09-26 07:30:49 on dist-test-slave-wdjc"
server_key: "9461b7dff4e6c6e546144ff23ecfef2e"
server_key_iv: "624e57468f66a312b013430d29c53acf"
server_key_version: "encryptionkey@0"
I20260926 07:30:49.926796 725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407783529717-22743-0/minicluster-data/ts-1/wal/instance:
uuid: "225f68f5a3334808b45a1756b3734718"
format_stamp: "Formatted at 2026-09-26 07:30:49 on dist-test-slave-wdjc"
server_key: "9461b7dff4e6c6e546144ff23ecfef2e"
server_key_iv: "624e57468f66a312b013430d29c53acf"
server_key_version: "encryptionkey@0"
I20260926 07:30:49.931794 725 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260926 07:30:49.935271 740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:49.937161 725 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 07:30:49.937352 725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407783529717-22743-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407783529717-22743-0/minicluster-data/ts-1/wal
uuid: "225f68f5a3334808b45a1756b3734718"
format_stamp: "Formatted at 2026-09-26 07:30:49 on dist-test-slave-wdjc"
server_key: "9461b7dff4e6c6e546144ff23ecfef2e"
server_key_iv: "624e57468f66a312b013430d29c53acf"
server_key_version: "encryptionkey@0"
I20260926 07:30:49.937505 725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407783529717-22743-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407783529717-22743-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407783529717-22743-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:30:49.954946 725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:30:49.959013 725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:30:49.959290 725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:30:49.960131 725 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:30:49.961207 725 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:30:49.961277 725 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:49.961331 725 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:30:49.961352 725 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:49.971990 725 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.194:35045
I20260926 07:30:49.972162 853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.194:35045 every 8 connection(s)
I20260926 07:30:49.973203 725 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407783529717-22743-0/minicluster-data/ts-1/data/info.pb
I20260926 07:30:49.973764 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 725
I20260926 07:30:49.973920 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407783529717-22743-0/minicluster-data/ts-1/wal/instance
I20260926 07:30:49.974282 22743 external_mini_cluster.cc:1428] Setting key be4b9df5decceccf6c3e65d814e5c504
WARNING: no policy specified for kudu/127.22.53.195@KRBTEST.COM; defaulting to no policy
Sep 26 07:30:49 dist-test-slave-wdjc krb5kdc[484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407849, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.194@KRBTEST.COM for kudu/127.22.53.254@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.53.195@KRBTEST.COM" created.
I20260926 07:30:49.989998 568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:49.975562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.194:42519 (local address 127.22.53.254:46439)
0926 07:30:49.975715 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:49.975720 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:49.976992 (+ 1272us) server_negotiation.cc:407] Connection header received
0926 07:30:49.978651 (+ 1659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:49.978655 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:49.978717 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:49.978831 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:49.980193 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:49.980899 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:49.982159 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:49.982406 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:49.982527 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:49.985512 (+ 2985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:49.985516 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:49.985518 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:49.985546 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:49.987181 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:49.988006 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:49.988011 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:49.988014 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:49.988126 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:49.988594 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:49.988601 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:49.988607 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:49.988921 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:49.989045 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:49.989653 (+ 608us) server_negotiation.cc:299] Negotiation successful
0926 07:30:49.989813 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 07:30:49.991014 856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:49.976170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:46439 (local address 127.22.53.194:42519)
0926 07:30:49.976837 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:49.976886 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:49.978375 (+ 1489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:49.979064 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:49.979072 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:49.979475 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:49.980016 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:49.980032 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:49.981051 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:49.981056 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:49.982001 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:49.982011 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:49.982581 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:49.982586 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:49.982771 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:49.983484 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:49.983502 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:49.985293 (+ 1791us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:30:49.987387 (+ 2094us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:49.987393 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:49.987427 (+ 34us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:49.987872 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:49.988242 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:49.988244 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:49.988245 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:49.988422 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:49.989056 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:30:49.989062 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:30:49.989346 (+ 284us) client_negotiation.cc:769] Sending connection context
0926 07:30:49.989601 (+ 255us) client_negotiation.cc:240] Negotiation successful
0926 07:30:49.990009 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":190,"threads_started":1}
I20260926 07:30:49.992460 854 heartbeater.cc:347] Connected to a master server at 127.22.53.254:46439
I20260926 07:30:49.992771 854 heartbeater.cc:464] Registering TS with master...
I20260926 07:30:49.993285 854 heartbeater.cc:511] Master 127.22.53.254:46439 requested a full tablet report, sending...
I20260926 07:30:49.994547 529 ts_manager.cc:194] Registered new tserver with Master: 225f68f5a3334808b45a1756b3734718 (127.22.53.194:35045)
I20260926 07:30:49.995189 529 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.53.194@KRBTEST.COM'} at 127.22.53.194:42519
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.53.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.195.keytab.
Entry for principal kudu/127.22.53.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.195.keytab.
I20260926 07:30:50.007346 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407783529717-22743-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407783529717-22743-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407783529717-22743-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407783529717-22743-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.53.195:0
--local_ip_for_outbound_sockets=127.22.53.195
--webserver_interface=127.22.53.195
--webserver_port=0
--tserver_master_addrs=127.22.53.254:46439
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.195.keytab
--principal=kudu/127.22.53.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:42697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:30:50.126338 859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:30:50.126636 859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:30:50.126681 859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:30:50.130962 859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:30:50.131057 859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:30:50.131122 859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.53.195
I20260926 07:30:50.136905 859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:42697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407783529717-22743-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407783529717-22743-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.195.keytab
--principal=kudu/127.22.53.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.53.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407783529717-22743-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.53.195
--webserver_port=0
--tserver_master_addrs=127.22.53.254:46439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.53.195
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407783529717-22743-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:30:50.138049 859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:30:50.139074 859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:30:50.149595 865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:50.149756 859 server_base.cc:1034] running on GCE node
W20260926 07:30:50.149658 867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:30:50.149595 864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:30:50.150486 859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:30:50.151209 859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:30:50.152382 859 hybrid_clock.cc:648] HybridClock initialized: now 1790407850152365 us; error 32 us; skew 500 ppm
Sep 26 07:30:50 dist-test-slave-wdjc krb5kdc[484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407850, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:30:50.155458 859 init.cc:377] Logged in from keytab as kudu/127.22.53.195@KRBTEST.COM (short username kudu)
I20260926 07:30:50.156487 859 webserver.cc:460] Webserver started at http://127.22.53.195:41481/ using document root <none> and password file <none>
I20260926 07:30:50.157135 859 fs_manager.cc:359] Metadata directory not provided
I20260926 07:30:50.157200 859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:30:50.157377 859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:30:50.159735 859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407783529717-22743-0/minicluster-data/ts-2/data/instance:
uuid: "7a92fd638749407f9e5018fbcb2a3747"
format_stamp: "Formatted at 2026-09-26 07:30:50 on dist-test-slave-wdjc"
server_key: "2e0300260fbebc1788d5ccd15132fdcd"
server_key_iv: "d5ee44b4d484a30b9d1f1dd4f012843f"
server_key_version: "encryptionkey@0"
I20260926 07:30:50.160315 859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407783529717-22743-0/minicluster-data/ts-2/wal/instance:
uuid: "7a92fd638749407f9e5018fbcb2a3747"
format_stamp: "Formatted at 2026-09-26 07:30:50 on dist-test-slave-wdjc"
server_key: "2e0300260fbebc1788d5ccd15132fdcd"
server_key_iv: "d5ee44b4d484a30b9d1f1dd4f012843f"
server_key_version: "encryptionkey@0"
I20260926 07:30:50.164391 859 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 07:30:50.167121 874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:50.168442 859 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 07:30:50.168620 859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407783529717-22743-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407783529717-22743-0/minicluster-data/ts-2/wal
uuid: "7a92fd638749407f9e5018fbcb2a3747"
format_stamp: "Formatted at 2026-09-26 07:30:50 on dist-test-slave-wdjc"
server_key: "2e0300260fbebc1788d5ccd15132fdcd"
server_key_iv: "d5ee44b4d484a30b9d1f1dd4f012843f"
server_key_version: "encryptionkey@0"
I20260926 07:30:50.168730 859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407783529717-22743-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407783529717-22743-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407783529717-22743-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:30:50.202742 859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:30:50.206532 859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:30:50.206704 859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:30:50.207288 859 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:30:50.208153 859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:30:50.208204 859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:50.208246 859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:30:50.208276 859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:30:50.217932 859 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.195:39241
I20260926 07:30:50.217955 987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.195:39241 every 8 connection(s)
I20260926 07:30:50.218981 859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407783529717-22743-0/minicluster-data/ts-2/data/info.pb
I20260926 07:30:50.225217 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 859
I20260926 07:30:50.225405 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407783529717-22743-0/minicluster-data/ts-2/wal/instance
I20260926 07:30:50.225800 22743 external_mini_cluster.cc:1428] Setting key 04292a0c2594963da2ffe6fb7b18d7e7
Sep 26 07:30:50 dist-test-slave-wdjc krb5kdc[484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407850, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.195@KRBTEST.COM for kudu/127.22.53.254@KRBTEST.COM
I20260926 07:30:50.236682 568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:50.221048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.195:49121 (local address 127.22.53.254:46439)
0926 07:30:50.221194 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:50.221197 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:50.222056 (+ 859us) server_negotiation.cc:407] Connection header received
0926 07:30:50.224112 (+ 2056us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:50.224118 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:50.224212 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:50.224354 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:50.226426 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:50.227344 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:50.229192 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:50.229526 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:50.229598 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:50.232670 (+ 3072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:50.232674 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:50.232676 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:50.232704 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:50.234348 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:50.234952 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:50.234955 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:50.234956 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:50.235038 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:50.235455 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:50.235458 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:50.235460 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:50.235678 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:50.235771 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:50.236317 (+ 546us) server_negotiation.cc:299] Negotiation successful
0926 07:30:50.236447 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 07:30:50.237768 990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:50.221368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:46439 (local address 127.22.53.195:49121)
0926 07:30:50.221883 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:50.221930 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:50.223719 (+ 1789us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:50.224652 (+ 933us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:50.224666 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:50.225329 (+ 663us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:30:50.226177 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:50.226204 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:50.227545 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:50.227550 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:50.229003 (+ 1453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:50.229016 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:50.229798 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:50.229800 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:50.229969 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:50.230693 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:30:50.230712 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:30:50.232400 (+ 1688us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:30:50.234499 (+ 2099us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:50.234505 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:50.234521 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:50.234837 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:50.235155 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:30:50.235158 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:30:50.235160 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:30:50.235327 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:30:50.235787 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:30:50.235793 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:30:50.236075 (+ 282us) client_negotiation.cc:769] Sending connection context
0926 07:30:50.236248 (+ 173us) client_negotiation.cc:240] Negotiation successful
0926 07:30:50.236654 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":132,"threads_started":1}
I20260926 07:30:50.239215 988 heartbeater.cc:347] Connected to a master server at 127.22.53.254:46439
I20260926 07:30:50.239526 988 heartbeater.cc:464] Registering TS with master...
I20260926 07:30:50.240161 988 heartbeater.cc:511] Master 127.22.53.254:46439 requested a full tablet report, sending...
I20260926 07:30:50.241410 529 ts_manager.cc:194] Registered new tserver with Master: 7a92fd638749407f9e5018fbcb2a3747 (127.22.53.195:39241)
I20260926 07:30:50.242044 529 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.53.195@KRBTEST.COM'} at 127.22.53.195:49121
I20260926 07:30:50.252780 22743 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:30:50.262504 568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:50.254897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53798 (local address 127.22.53.254:46439)
0926 07:30:50.255107 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:50.255110 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:50.255127 (+ 17us) server_negotiation.cc:407] Connection header received
0926 07:30:50.255257 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:50.255259 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:50.255303 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:50.255372 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:30:50.256115 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:50.256706 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:50.257570 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:50.257781 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:50.257861 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:50.259356 (+ 1495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:30:50.259359 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:30:50.259361 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:30:50.259384 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:30:50.260825 (+ 1441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:50.261252 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:50.261255 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:50.261257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:50.261361 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:30:50.261605 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:30:50.261608 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:30:50.261609 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:30:50.261885 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:30:50.261994 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:50.262186 (+ 192us) server_negotiation.cc:299] Negotiation successful
0926 07:30:50.262292 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260926 07:30:50.265575 529 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53798:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:30:50.268778 529 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:30:50.284703 999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:50.277852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.194:35045 (local address 127.0.0.1:55426)
0926 07:30:50.278784 (+ 932us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:50.278800 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:50.278952 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:50.279811 (+ 859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:50.279848 (+ 37us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:50.279875 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:30:50.280114 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:50.280123 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:50.281677 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:50.281680 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:50.283039 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:50.283049 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:50.284062 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:50.284066 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:50.284179 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:50.284241 (+ 62us) client_negotiation.cc:769] Sending connection context
0926 07:30:50.284356 (+ 115us) client_negotiation.cc:240] Negotiation successful
0926 07:30:50.284501 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":838,"mutex_wait_us":351,"spinlock_wait_cycles":12800,"thread_start_us":89,"threads_started":1}
I20260926 07:30:50.284854 1003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:50.278608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.193:42315 (local address 127.0.0.1:46160)
0926 07:30:50.278832 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:50.278846 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:50.278956 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:50.280810 (+ 1854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:50.280813 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:50.280831 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:30:50.280901 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:50.280909 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:50.282207 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:50.282210 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:50.283414 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:50.283426 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:50.284295 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:50.284299 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:50.284402 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:50.284426 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 07:30:50.284514 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 07:30:50.284617 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"mutex_wait_us":7,"spinlock_wait_cycles":37248,"thread_start_us":99,"threads_started":1}
I20260926 07:30:50.285095 1004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:50.279880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46160 (local address 127.22.53.193:42315)
0926 07:30:50.280245 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:50.280251 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:50.280271 (+ 20us) server_negotiation.cc:407] Connection header received
0926 07:30:50.280343 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:50.280347 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:50.280585 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:50.280727 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:30:50.281031 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:50.282089 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:50.283550 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:50.284164 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:50.284277 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:50.284428 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:50.284574 (+ 146us) server_negotiation.cc:299] Negotiation successful
0926 07:30:50.284683 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":150,"threads_started":1}
I20260926 07:30:50.285641 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:50.278127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55426 (local address 127.22.53.194:35045)
0926 07:30:50.279280 (+ 1153us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:50.279293 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:50.279319 (+ 26us) server_negotiation.cc:407] Connection header received
0926 07:30:50.279402 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:50.279408 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:50.279635 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:50.279802 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:30:50.280255 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:50.281518 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:50.283185 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:50.283927 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:50.284020 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:50.284136 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:50.285103 (+ 967us) server_negotiation.cc:299] Negotiation successful
0926 07:30:50.285468 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1019,"thread_start_us":125,"threads_started":1}
I20260926 07:30:50.287936 654 tablet_service.cc:1467] Processing CreateTablet for tablet 9f6df8e8b37b4dd7b15aa962b30736ad (DEFAULT_TABLE table=test-table [id=5cd4d336c730450190dbf42331a89640]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:30:50.287936 788 tablet_service.cc:1467] Processing CreateTablet for tablet 9f6df8e8b37b4dd7b15aa962b30736ad (DEFAULT_TABLE table=test-table [id=5cd4d336c730450190dbf42331a89640]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:30:50.289036 788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f6df8e8b37b4dd7b15aa962b30736ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:30:50.289052 654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f6df8e8b37b4dd7b15aa962b30736ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:30:50.289870 1000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:50.277852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.195:39241 (local address 127.0.0.1:51562)
0926 07:30:50.278742 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:50.278798 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:50.278951 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:50.280607 (+ 1656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:50.280610 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:50.280628 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:30:50.280734 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:50.280740 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:50.282105 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:50.282108 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:50.284444 (+ 2336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:50.284453 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:50.289365 (+ 4912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:50.289370 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:50.289492 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:50.289525 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 07:30:50.289604 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 07:30:50.289703 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":698,"mutex_wait_us":1090,"spinlock_wait_cycles":3328,"thread_start_us":101,"threads_started":1}
I20260926 07:30:50.290894 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:50.278015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51562 (local address 127.22.53.195:39241)
0926 07:30:50.280163 (+ 2148us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:50.280177 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:50.280201 (+ 24us) server_negotiation.cc:407] Connection header received
0926 07:30:50.280278 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:50.280283 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:50.280451 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:50.280634 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:30:50.280865 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:50.281687 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:50.284608 (+ 2921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:50.289158 (+ 4550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:50.289304 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:50.290549 (+ 1245us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:50.290618 (+ 69us) server_negotiation.cc:299] Negotiation successful
0926 07:30:50.290734 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2032,"thread_start_us":179,"threads_started":1}
I20260926 07:30:50.294234 922 tablet_service.cc:1467] Processing CreateTablet for tablet 9f6df8e8b37b4dd7b15aa962b30736ad (DEFAULT_TABLE table=test-table [id=5cd4d336c730450190dbf42331a89640]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:30:50.295609 922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f6df8e8b37b4dd7b15aa962b30736ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:30:50.296277 1005 tablet_bootstrap.cc:492] T 9f6df8e8b37b4dd7b15aa962b30736ad P 3eeed72654284329a7b14dc24092d366: Bootstrap starting.
I20260926 07:30:50.297878 1006 tablet_bootstrap.cc:492] T 9f6df8e8b37b4dd7b15aa962b30736ad P 225f68f5a3334808b45a1756b3734718: Bootstrap starting.
I20260926 07:30:50.298977 1005 tablet_bootstrap.cc:654] T 9f6df8e8b37b4dd7b15aa962b30736ad P 3eeed72654284329a7b14dc24092d366: Neither blocks nor log segments found. Creating new log.
I20260926 07:30:50.299921 1005 log.cc:824] T 9f6df8e8b37b4dd7b15aa962b30736ad P 3eeed72654284329a7b14dc24092d366: Log is configured to *not* fsync() on all Append() calls
I20260926 07:30:50.300833 1006 tablet_bootstrap.cc:654] T 9f6df8e8b37b4dd7b15aa962b30736ad P 225f68f5a3334808b45a1756b3734718: Neither blocks nor log segments found. Creating new log.
I20260926 07:30:50.301728 1006 log.cc:824] T 9f6df8e8b37b4dd7b15aa962b30736ad P 225f68f5a3334808b45a1756b3734718: Log is configured to *not* fsync() on all Append() calls
I20260926 07:30:50.302502 1005 tablet_bootstrap.cc:492] T 9f6df8e8b37b4dd7b15aa962b30736ad P 3eeed72654284329a7b14dc24092d366: No bootstrap required, opened a new log
I20260926 07:30:50.302753 1005 ts_tablet_manager.cc:1397] T 9f6df8e8b37b4dd7b15aa962b30736ad P 3eeed72654284329a7b14dc24092d366: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260926 07:30:50.303259 1009 tablet_bootstrap.cc:492] T 9f6df8e8b37b4dd7b15aa962b30736ad P 7a92fd638749407f9e5018fbcb2a3747: Bootstrap starting.
I20260926 07:30:50.303982 1006 tablet_bootstrap.cc:492] T 9f6df8e8b37b4dd7b15aa962b30736ad P 225f68f5a3334808b45a1756b3734718: No bootstrap required, opened a new log
I20260926 07:30:50.304193 1006 ts_tablet_manager.cc:1397] T 9f6df8e8b37b4dd7b15aa962b30736ad P 225f68f5a3334808b45a1756b3734718: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 07:30:50.306044 1009 tablet_bootstrap.cc:654] T 9f6df8e8b37b4dd7b15aa962b30736ad P 7a92fd638749407f9e5018fbcb2a3747: Neither blocks nor log segments found. Creating new log.
I20260926 07:30:50.306998 1009 log.cc:824] T 9f6df8e8b37b4dd7b15aa962b30736ad P 7a92fd638749407f9e5018fbcb2a3747: Log is configured to *not* fsync() on all Append() calls
I20260926 07:30:50.308300 1006 raft_consensus.cc:348] T 9f6df8e8b37b4dd7b15aa962b30736ad P 225f68f5a3334808b45a1756b3734718 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a92fd638749407f9e5018fbcb2a3747" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 39241 } } peers { permanent_uuid: "225f68f5a3334808b45a1756b3734718" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 35045 } } peers { permanent_uuid: "3eeed72654284329a7b14dc24092d366" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 42315 } }
I20260926 07:30:50.308225 1005 raft_consensus.cc:348] T 9f6df8e8b37b4dd7b15aa962b30736ad P 3eeed72654284329a7b14dc24092d366 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a92fd638749407f9e5018fbcb2a3747" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 39241 } } peers { permanent_uuid: "225f68f5a3334808b45a1756b3734718" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 35045 } } peers { permanent_uuid: "3eeed72654284329a7b14dc24092d366" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 42315 } }
I20260926 07:30:50.308571 1005 raft_consensus.cc:374] T 9f6df8e8b37b4dd7b15aa962b30736ad P 3eeed72654284329a7b14dc24092d366 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:30:50.308566 1006 raft_consensus.cc:374] T 9f6df8e8b37b4dd7b15aa962b30736ad P 225f68f5a3334808b45a1756b3734718 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:30:50.308629 1005 raft_consensus.cc:729] T 9f6df8e8b37b4dd7b15aa962b30736ad P 3eeed72654284329a7b14dc24092d366 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3eeed72654284329a7b14dc24092d366, State: Initialized, Role: FOLLOWER
I20260926 07:30:50.308629 1006 raft_consensus.cc:729] T 9f6df8e8b37b4dd7b15aa962b30736ad P 225f68f5a3334808b45a1756b3734718 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 225f68f5a3334808b45a1756b3734718, State: Initialized, Role: FOLLOWER
I20260926 07:30:50.309176 1006 consensus_queue.cc:260] T 9f6df8e8b37b4dd7b15aa962b30736ad P 225f68f5a3334808b45a1756b3734718 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a92fd638749407f9e5018fbcb2a3747" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 39241 } } peers { permanent_uuid: "225f68f5a3334808b45a1756b3734718" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 35045 } } peers { permanent_uuid: "3eeed72654284329a7b14dc24092d366" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 42315 } }
I20260926 07:30:50.309175 1005 consensus_queue.cc:260] T 9f6df8e8b37b4dd7b15aa962b30736ad P 3eeed72654284329a7b14dc24092d366 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a92fd638749407f9e5018fbcb2a3747" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 39241 } } peers { permanent_uuid: "225f68f5a3334808b45a1756b3734718" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 35045 } } peers { permanent_uuid: "3eeed72654284329a7b14dc24092d366" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 42315 } }
I20260926 07:30:50.309685 1009 tablet_bootstrap.cc:492] T 9f6df8e8b37b4dd7b15aa962b30736ad P 7a92fd638749407f9e5018fbcb2a3747: No bootstrap required, opened a new log
I20260926 07:30:50.309930 1009 ts_tablet_manager.cc:1397] T 9f6df8e8b37b4dd7b15aa962b30736ad P 7a92fd638749407f9e5018fbcb2a3747: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260926 07:30:50.310314 854 heartbeater.cc:503] Master 127.22.53.254:46439 was elected leader, sending a full tablet report...
I20260926 07:30:50.310395 720 heartbeater.cc:503] Master 127.22.53.254:46439 was elected leader, sending a full tablet report...
I20260926 07:30:50.311560 1006 ts_tablet_manager.cc:1428] T 9f6df8e8b37b4dd7b15aa962b30736ad P 225f68f5a3334808b45a1756b3734718: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260926 07:30:50.311584 1005 ts_tablet_manager.cc:1428] T 9f6df8e8b37b4dd7b15aa962b30736ad P 3eeed72654284329a7b14dc24092d366: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
I20260926 07:30:50.315626 1009 raft_consensus.cc:348] T 9f6df8e8b37b4dd7b15aa962b30736ad P 7a92fd638749407f9e5018fbcb2a3747 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a92fd638749407f9e5018fbcb2a3747" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 39241 } } peers { permanent_uuid: "225f68f5a3334808b45a1756b3734718" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 35045 } } peers { permanent_uuid: "3eeed72654284329a7b14dc24092d366" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 42315 } }
I20260926 07:30:50.315997 1009 raft_consensus.cc:374] T 9f6df8e8b37b4dd7b15aa962b30736ad P 7a92fd638749407f9e5018fbcb2a3747 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:30:50.316066 1009 raft_consensus.cc:729] T 9f6df8e8b37b4dd7b15aa962b30736ad P 7a92fd638749407f9e5018fbcb2a3747 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a92fd638749407f9e5018fbcb2a3747, State: Initialized, Role: FOLLOWER
I20260926 07:30:50.316521 1009 consensus_queue.cc:260] T 9f6df8e8b37b4dd7b15aa962b30736ad P 7a92fd638749407f9e5018fbcb2a3747 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a92fd638749407f9e5018fbcb2a3747" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 39241 } } peers { permanent_uuid: "225f68f5a3334808b45a1756b3734718" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 35045 } } peers { permanent_uuid: "3eeed72654284329a7b14dc24092d366" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 42315 } }
I20260926 07:30:50.317395 988 heartbeater.cc:503] Master 127.22.53.254:46439 was elected leader, sending a full tablet report...
I20260926 07:30:50.317817 1009 ts_tablet_manager.cc:1428] T 9f6df8e8b37b4dd7b15aa962b30736ad P 7a92fd638749407f9e5018fbcb2a3747: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260926 07:30:50.352715 1012 raft_consensus.cc:482] T 9f6df8e8b37b4dd7b15aa962b30736ad P 225f68f5a3334808b45a1756b3734718 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:30:50.353030 1012 raft_consensus.cc:504] T 9f6df8e8b37b4dd7b15aa962b30736ad P 225f68f5a3334808b45a1756b3734718 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a92fd638749407f9e5018fbcb2a3747" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 39241 } } peers { permanent_uuid: "225f68f5a3334808b45a1756b3734718" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 35045 } } peers { permanent_uuid: "3eeed72654284329a7b14dc24092d366" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 42315 } }
I20260926 07:30:50.354403 1012 leader_election.cc:288] T 9f6df8e8b37b4dd7b15aa962b30736ad P 225f68f5a3334808b45a1756b3734718 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a92fd638749407f9e5018fbcb2a3747 (127.22.53.195:39241), 3eeed72654284329a7b14dc24092d366 (127.22.53.193:42315)
I20260926 07:30:50.359267 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:50.354914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.194:39527 (local address 127.22.53.195:39241)
0926 07:30:50.355065 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:50.355071 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:50.355087 (+ 16us) server_negotiation.cc:407] Connection header received
0926 07:30:50.355234 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:50.355239 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:50.355294 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:50.355374 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:30:50.355713 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:50.356491 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:50.357908 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:50.358602 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:50.358756 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:50.358799 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:50.359042 (+ 243us) server_negotiation.cc:299] Negotiation successful
0926 07:30:50.359119 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 07:30:50.359267 856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:50.354752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.195:39241 (local address 127.22.53.194:39527)
0926 07:30:50.354995 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:50.355013 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:50.355133 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:50.355415 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:50.355419 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:50.355445 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:30:50.355599 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:50.355610 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:50.356649 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:50.356653 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:50.357787 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:50.357797 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:50.358761 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:50.358765 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:50.358886 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:50.358923 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 07:30:50.358997 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 07:30:50.359088 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119}
I20260926 07:30:50.359452 1004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:50.355209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.194:37873 (local address 127.22.53.193:42315)
0926 07:30:50.355344 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:50.355350 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:50.355582 (+ 232us) server_negotiation.cc:407] Connection header received
0926 07:30:50.355699 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:50.355702 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:50.355754 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:50.355833 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:30:50.356159 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:50.356872 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:50.358211 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:50.358836 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:50.358967 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:50.359027 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:50.359225 (+ 198us) server_negotiation.cc:299] Negotiation successful
0926 07:30:50.359324 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 07:30:50.359745 1014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:50.355037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.193:42315 (local address 127.22.53.194:37873)
0926 07:30:50.355467 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:50.355481 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:30:50.355581 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:30:50.355911 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:30:50.355914 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:30:50.355932 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:30:50.356029 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:50.356034 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:50.356989 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:50.356993 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:50.358086 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:30:50.358096 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:30:50.358953 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:30:50.358955 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:30:50.359089 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:50.359111 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 07:30:50.359189 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 07:30:50.359255 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":122,"threads_started":1}
I20260926 07:30:50.360149 942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f6df8e8b37b4dd7b15aa962b30736ad" candidate_uuid: "225f68f5a3334808b45a1756b3734718" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a92fd638749407f9e5018fbcb2a3747" is_pre_election: true
I20260926 07:30:50.360497 942 raft_consensus.cc:2454] T 9f6df8e8b37b4dd7b15aa962b30736ad P 7a92fd638749407f9e5018fbcb2a3747 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 225f68f5a3334808b45a1756b3734718 in term 0.
I20260926 07:30:50.361042 741 leader_election.cc:302] T 9f6df8e8b37b4dd7b15aa962b30736ad P 225f68f5a3334808b45a1756b3734718 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 225f68f5a3334808b45a1756b3734718, 7a92fd638749407f9e5018fbcb2a3747; no voters:
I20260926 07:30:50.361091 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f6df8e8b37b4dd7b15aa962b30736ad" candidate_uuid: "225f68f5a3334808b45a1756b3734718" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eeed72654284329a7b14dc24092d366" is_pre_election: true
I20260926 07:30:50.361310 1012 raft_consensus.cc:2781] T 9f6df8e8b37b4dd7b15aa962b30736ad P 225f68f5a3334808b45a1756b3734718 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:30:50.361399 1012 raft_consensus.cc:482] T 9f6df8e8b37b4dd7b15aa962b30736ad P 225f68f5a3334808b45a1756b3734718 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:30:50.361431 1012 raft_consensus.cc:3037] T 9f6df8e8b37b4dd7b15aa962b30736ad P 225f68f5a3334808b45a1756b3734718 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:30:50.361380 674 raft_consensus.cc:2454] T 9f6df8e8b37b4dd7b15aa962b30736ad P 3eeed72654284329a7b14dc24092d366 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 225f68f5a3334808b45a1756b3734718 in term 0.
I20260926 07:30:50.362661 1012 raft_consensus.cc:504] T 9f6df8e8b37b4dd7b15aa962b30736ad P 225f68f5a3334808b45a1756b3734718 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a92fd638749407f9e5018fbcb2a3747" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 39241 } } peers { permanent_uuid: "225f68f5a3334808b45a1756b3734718" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 35045 } } peers { permanent_uuid: "3eeed72654284329a7b14dc24092d366" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 42315 } }
I20260926 07:30:50.363071 1012 leader_election.cc:288] T 9f6df8e8b37b4dd7b15aa962b30736ad P 225f68f5a3334808b45a1756b3734718 [CANDIDATE]: Term 1 election: Requested vote from peers 7a92fd638749407f9e5018fbcb2a3747 (127.22.53.195:39241), 3eeed72654284329a7b14dc24092d366 (127.22.53.193:42315)
I20260926 07:30:50.363516 942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f6df8e8b37b4dd7b15aa962b30736ad" candidate_uuid: "225f68f5a3334808b45a1756b3734718" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a92fd638749407f9e5018fbcb2a3747"
I20260926 07:30:50.363545 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f6df8e8b37b4dd7b15aa962b30736ad" candidate_uuid: "225f68f5a3334808b45a1756b3734718" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eeed72654284329a7b14dc24092d366"
I20260926 07:30:50.363660 674 raft_consensus.cc:3037] T 9f6df8e8b37b4dd7b15aa962b30736ad P 3eeed72654284329a7b14dc24092d366 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:30:50.363660 942 raft_consensus.cc:3037] T 9f6df8e8b37b4dd7b15aa962b30736ad P 7a92fd638749407f9e5018fbcb2a3747 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:30:50.365096 674 raft_consensus.cc:2454] T 9f6df8e8b37b4dd7b15aa962b30736ad P 3eeed72654284329a7b14dc24092d366 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 225f68f5a3334808b45a1756b3734718 in term 1.
I20260926 07:30:50.365096 942 raft_consensus.cc:2454] T 9f6df8e8b37b4dd7b15aa962b30736ad P 7a92fd638749407f9e5018fbcb2a3747 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 225f68f5a3334808b45a1756b3734718 in term 1.
I20260926 07:30:50.365438 741 leader_election.cc:302] T 9f6df8e8b37b4dd7b15aa962b30736ad P 225f68f5a3334808b45a1756b3734718 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 225f68f5a3334808b45a1756b3734718, 3eeed72654284329a7b14dc24092d366; no voters:
I20260926 07:30:50.365635 1012 raft_consensus.cc:2781] T 9f6df8e8b37b4dd7b15aa962b30736ad P 225f68f5a3334808b45a1756b3734718 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:30:50.365880 1012 raft_consensus.cc:686] T 9f6df8e8b37b4dd7b15aa962b30736ad P 225f68f5a3334808b45a1756b3734718 [term 1 LEADER]: Becoming Leader. State: Replica: 225f68f5a3334808b45a1756b3734718, State: Running, Role: LEADER
I20260926 07:30:50.366267 1012 consensus_queue.cc:237] T 9f6df8e8b37b4dd7b15aa962b30736ad P 225f68f5a3334808b45a1756b3734718 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a92fd638749407f9e5018fbcb2a3747" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 39241 } } peers { permanent_uuid: "225f68f5a3334808b45a1756b3734718" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 35045 } } peers { permanent_uuid: "3eeed72654284329a7b14dc24092d366" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 42315 } }
I20260926 07:30:50.369709 529 catalog_manager.cc:5495] T 9f6df8e8b37b4dd7b15aa962b30736ad P 225f68f5a3334808b45a1756b3734718 reported cstate change: term changed from 0 to 1, leader changed from <none> to 225f68f5a3334808b45a1756b3734718 (127.22.53.194). New cstate: current_term: 1 leader_uuid: "225f68f5a3334808b45a1756b3734718" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a92fd638749407f9e5018fbcb2a3747" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 39241 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "225f68f5a3334808b45a1756b3734718" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 35045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3eeed72654284329a7b14dc24092d366" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 42315 } health_report { overall_health: UNKNOWN } } }
I20260926 07:30:50.385491 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:50.381615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55430 (local address 127.22.53.194:35045)
0926 07:30:50.381759 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 07:30:50.381763 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:30:50.381780 (+ 17us) server_negotiation.cc:407] Connection header received
0926 07:30:50.381983 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:30:50.381986 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:30:50.382052 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:30:50.382182 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:30:50.382446 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:50.382957 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:50.383849 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:30:50.384067 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:30:50.384179 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:30:50.384500 (+ 321us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:30:50.385038 (+ 538us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:30:50.385174 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 07:30:50.385237 (+ 63us) server_negotiation.cc:299] Negotiation successful
0926 07:30:50.385312 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 07:30:50.392307 674 raft_consensus.cc:1261] T 9f6df8e8b37b4dd7b15aa962b30736ad P 3eeed72654284329a7b14dc24092d366 [term 1 FOLLOWER]: Refusing update from remote peer 225f68f5a3334808b45a1756b3734718: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:30:50.392305 942 raft_consensus.cc:1261] T 9f6df8e8b37b4dd7b15aa962b30736ad P 7a92fd638749407f9e5018fbcb2a3747 [term 1 FOLLOWER]: Refusing update from remote peer 225f68f5a3334808b45a1756b3734718: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:30:50.393232 1012 consensus_queue.cc:1035] T 9f6df8e8b37b4dd7b15aa962b30736ad P 225f68f5a3334808b45a1756b3734718 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a92fd638749407f9e5018fbcb2a3747" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 39241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:30:50.393544 1015 consensus_queue.cc:1035] T 9f6df8e8b37b4dd7b15aa962b30736ad P 225f68f5a3334808b45a1756b3734718 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eeed72654284329a7b14dc24092d366" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:30:50.402415 1020 mvcc.cc:205] Tried to move back new op lower bound from 7333510555197706240 to 7333510555103047680. Current Snapshot: MvccSnapshot[applied={T|T < 7333510555197706240}]
I20260926 07:30:50.402815 1019 mvcc.cc:205] Tried to move back new op lower bound from 7333510555197706240 to 7333510555103047680. Current Snapshot: MvccSnapshot[applied={T|T < 7333510555197706240}]
I20260926 07:30:50.411443 529 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53798:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:30:50.411703 529 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53798:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:30:50.414399 529 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 500d83d94ada4846b152ca2d5f9f2f51: Sending DeleteTablet for 3 replicas of tablet 9f6df8e8b37b4dd7b15aa962b30736ad
I20260926 07:30:50.415308 788 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f6df8e8b37b4dd7b15aa962b30736ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:30:50 UTC) from {username='kudu', principal='kudu/127.22.53.254@KRBTEST.COM'} at 127.0.0.1:55426
I20260926 07:30:50.415306 922 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f6df8e8b37b4dd7b15aa962b30736ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:30:50 UTC) from {username='kudu', principal='kudu/127.22.53.254@KRBTEST.COM'} at 127.0.0.1:51562
I20260926 07:30:50.415417 654 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f6df8e8b37b4dd7b15aa962b30736ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:30:50 UTC) from {username='kudu', principal='kudu/127.22.53.254@KRBTEST.COM'} at 127.0.0.1:46160
I20260926 07:30:50.415921 1028 tablet_replica.cc:330] T 9f6df8e8b37b4dd7b15aa962b30736ad P 7a92fd638749407f9e5018fbcb2a3747: stopping tablet replica
I20260926 07:30:50.416000 1030 tablet_replica.cc:330] T 9f6df8e8b37b4dd7b15aa962b30736ad P 3eeed72654284329a7b14dc24092d366: stopping tablet replica
I20260926 07:30:50.416309 1029 tablet_replica.cc:330] T 9f6df8e8b37b4dd7b15aa962b30736ad P 225f68f5a3334808b45a1756b3734718: stopping tablet replica
I20260926 07:30:50.416406 1030 raft_consensus.cc:2229] T 9f6df8e8b37b4dd7b15aa962b30736ad P 3eeed72654284329a7b14dc24092d366 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:30:50.416607 1029 raft_consensus.cc:2229] T 9f6df8e8b37b4dd7b15aa962b30736ad P 225f68f5a3334808b45a1756b3734718 [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:30:50.416831 1030 raft_consensus.cc:2258] T 9f6df8e8b37b4dd7b15aa962b30736ad P 3eeed72654284329a7b14dc24092d366 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:30:50.416898 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 591
I20260926 07:30:50.417992 1029 raft_consensus.cc:2258] T 9f6df8e8b37b4dd7b15aa962b30736ad P 225f68f5a3334808b45a1756b3734718 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:30:50.419646 1028 raft_consensus.cc:2229] T 9f6df8e8b37b4dd7b15aa962b30736ad P 7a92fd638749407f9e5018fbcb2a3747 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:30:50.420083 1028 raft_consensus.cc:2258] T 9f6df8e8b37b4dd7b15aa962b30736ad P 7a92fd638749407f9e5018fbcb2a3747 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:30:50.420329 1029 ts_tablet_manager.cc:1905] T 9f6df8e8b37b4dd7b15aa962b30736ad P 225f68f5a3334808b45a1756b3734718: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:30:50.422062 1028 ts_tablet_manager.cc:1905] T 9f6df8e8b37b4dd7b15aa962b30736ad P 7a92fd638749407f9e5018fbcb2a3747: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:30:50.424146 1029 ts_tablet_manager.cc:1918] T 9f6df8e8b37b4dd7b15aa962b30736ad P 225f68f5a3334808b45a1756b3734718: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:30:50.424276 1029 log.cc:1196] T 9f6df8e8b37b4dd7b15aa962b30736ad P 225f68f5a3334808b45a1756b3734718: Deleting WAL directory at /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407783529717-22743-0/minicluster-data/ts-1/wal/wals/9f6df8e8b37b4dd7b15aa962b30736ad
I20260926 07:30:50.424678 1029 ts_tablet_manager.cc:1939] T 9f6df8e8b37b4dd7b15aa962b30736ad P 225f68f5a3334808b45a1756b3734718: Deleting consensus metadata
W20260926 07:30:50.424925 513 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.53.193:42315 (error 108)
I20260926 07:30:50.425298 1028 ts_tablet_manager.cc:1918] T 9f6df8e8b37b4dd7b15aa962b30736ad P 7a92fd638749407f9e5018fbcb2a3747: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:30:50.425467 1028 log.cc:1196] T 9f6df8e8b37b4dd7b15aa962b30736ad P 7a92fd638749407f9e5018fbcb2a3747: Deleting WAL directory at /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407783529717-22743-0/minicluster-data/ts-2/wal/wals/9f6df8e8b37b4dd7b15aa962b30736ad
I20260926 07:30:50.425860 1028 ts_tablet_manager.cc:1939] T 9f6df8e8b37b4dd7b15aa962b30736ad P 7a92fd638749407f9e5018fbcb2a3747: Deleting consensus metadata
I20260926 07:30:50.426105 514 catalog_manager.cc:4841] TS 225f68f5a3334808b45a1756b3734718 (127.22.53.194:35045): tablet 9f6df8e8b37b4dd7b15aa962b30736ad (table test-table [id=5cd4d336c730450190dbf42331a89640]) successfully deleted
I20260926 07:30:50.426563 1000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:30:50.426231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.193:42315 (local address 127.0.0.1:46170)
0926 07:30:50.426410 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 07:30:50.426486 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.193:42315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
W20260926 07:30:50.426795 513 catalog_manager.cc:4584] TS 3eeed72654284329a7b14dc24092d366 (127.22.53.193:42315): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9f6df8e8b37b4dd7b15aa962b30736ad: Network error: Client connection negotiation failed: client connection to 127.22.53.193:42315: connect: Connection refused (error 111)
I20260926 07:30:50.427187 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 725
I20260926 07:30:50.428030 513 catalog_manager.cc:4841] TS 7a92fd638749407f9e5018fbcb2a3747 (127.22.53.195:39241): tablet 9f6df8e8b37b4dd7b15aa962b30736ad (table test-table [id=5cd4d336c730450190dbf42331a89640]) successfully deleted
I20260926 07:30:50.432519 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 859
I20260926 07:30:50.439249 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 497
2026-09-26T07:30:50Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2381 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:30:50 dist-test-slave-wdjc krb5kdc[1034](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:30:50 dist-test-slave-wdjc krb5kdc[1034](info): set up 2 sockets
Sep 26 07:30:50 dist-test-slave-wdjc krb5kdc[1034](info): commencing operation
krb5kdc: starting...
W20260926 07:30:51.497649 22743 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/krb5kdc/test-user.keytab.
Sep 26 07:30:51 dist-test-slave-wdjc krb5kdc[1034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407851, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:30:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:30:51Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.22.53.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.22.53.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.22.53.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/krb5kdc/rangeradmin_127.22.53.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.22.53.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/krb5kdc/rangeradmin_127.22.53.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.22.53.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.22.53.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.22.53.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/krb5kdc/rangerlookup_127.22.53.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.22.53.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/krb5kdc/rangerlookup_127.22.53.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.22.53.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.22.53.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.53.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/krb5kdc/HTTP_127.22.53.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.22.53.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/krb5kdc/HTTP_127.22.53.212@KRBTEST.COM.keytab.
I20260926 07:30:51.687391 22743 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskdIE4X7/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/minicluster-data/postgres -l logfile start
2026-09-26 07:30:54.406 UTC [1064] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 07:30:54.407 UTC [1064] LOG: listening on IPv4 address "127.22.53.212", port 56067
2026-09-26 07:30:54.411 UTC [1064] LOG: listening on Unix socket "/tmp/.s.PGSQL.56067"
2026-09-26 07:30:54.433 UTC [1067] LOG: database system was shut down at 2026-09-26 07:30:52 UTC
2026-09-26 07:30:54.437 UTC [1064] LOG: database system is ready to accept connections
I20260926 07:30:55.385612 22743 mini_postgres.cc:96] Postgres bound to 56067
2026-09-26 07:30:55.392 UTC [1075] FATAL: database "slave" does not exist
127.22.53.212:56067 - accepting connections
I20260926 07:30:55.393038 22743 mini_ranger.cc:161] Starting Ranger...
I20260926 07:30:55.411105 22743 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 07:30:55.990290 22743 mini_ranger.cc:87] Created ranger Postgres database
I20260926 07:30:55.990401 22743 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/minicluster-data/ranger-admin
2026-09-26 07:30:56,295 [I] DB FLAVOR :POSTGRES
2026-09-26 07:30:56,296 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 07:30:56,296 [I] Checking connection..
2026-09-26 07:30:56,296 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdIE4X7/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdIE4X7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.53.212:56067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 07:30:56,551 [I] Checking connection passed.
2026-09-26 07:30:56,552 [I] --------- Verifying version history table ---------
2026-09-26 07:30:56,552 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdIE4X7/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdIE4X7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.53.212:56067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 07:30:56,813 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 07:30:56,813 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdIE4X7/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdIE4X7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.53.212:56067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 07:30:57,082 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 07:30:57,083 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 07:30:57,083 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdIE4X7/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdIE4X7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.53.212:56067/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskdIE4X7/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 07:30:57.340 UTC [1139] WARNING: there is no transaction in progress
2026-09-26 07:30:57,347 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 07:30:57,348 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdIE4X7/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdIE4X7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.53.212:56067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 07:30:57,627 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 07:30:57,627 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 07:30:57,627 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdIE4X7/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdIE4X7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.53.212:56067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 07:30:57,886 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdIE4X7/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdIE4X7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.53.212:56067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 07:30:58,151 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdIE4X7/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdIE4X7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.53.212:56067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-wdjc.c.gcp-upstream.internal','N') ;"
2026-09-26 07:30:58,424 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdIE4X7/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdIE4X7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.53.212:56067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 07:30:58,684 [I] Table x_portal_user does not exist in database ranger
2026-09-26 07:30:58,684 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdIE4X7/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdIE4X7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.53.212:56067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 07:30:58,954 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 07:30:58,954 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdIE4X7/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdIE4X7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.53.212:56067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 07:30:59,205 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdIE4X7/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdIE4X7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.53.212:56067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 07:30:59,465 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 07:30:59,466 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdIE4X7/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdIE4X7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.53.212:56067/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskdIE4X7/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 07:31:00.584 UTC [1247] WARNING: there is no transaction in progress
2026-09-26 07:31:00.603 UTC [1247] WARNING: there is no transaction in progress
2026-09-26 07:31:00.622 UTC [1247] WARNING: there is no transaction in progress
2026-09-26 07:31:00.641 UTC [1247] WARNING: there is no transaction in progress
2026-09-26 07:31:00.663 UTC [1247] WARNING: there is no transaction in progress
2026-09-26 07:31:00.682 UTC [1247] WARNING: there is no transaction in progress
2026-09-26 07:31:00.756 UTC [1247] WARNING: there is no transaction in progress
2026-09-26 07:31:00.765 UTC [1247] WARNING: there is no transaction in progress
2026-09-26 07:31:00.785 UTC [1247] WARNING: there is no transaction in progress
2026-09-26 07:31:00.798 UTC [1247] WARNING: there is no transaction in progress
2026-09-26 07:31:00.811 UTC [1247] WARNING: there is no transaction in progress
2026-09-26 07:31:00.825 UTC [1247] WARNING: there is no transaction in progress
2026-09-26 07:31:00.836 UTC [1247] WARNING: there is no transaction in progress
2026-09-26 07:31:00.844 UTC [1247] WARNING: there is no transaction in progress
2026-09-26 07:31:00.852 UTC [1247] WARNING: there is no transaction in progress
2026-09-26 07:31:00.860 UTC [1247] WARNING: there is no transaction in progress
2026-09-26 07:31:01,683 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 07:31:01,683 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdIE4X7/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdIE4X7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.53.212:56067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 07:31:01,955 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 07:31:01,955 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdIE4X7/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdIE4X7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.53.212:56067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 07:31:02,216 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 07:31:02,216 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdIE4X7/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdIE4X7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.53.212:56067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 07:31:02,476 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 07:31:02,477 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdIE4X7/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdIE4X7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.53.212:56067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 07:31:02,749 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdIE4X7/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdIE4X7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.53.212:56067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 07:31:03,024 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdIE4X7/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdIE4X7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.53.212:56067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-wdjc.c.gcp-upstream.internal';"
2026-09-26 07:31:03,291 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 07:31:03,292 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdIE4X7/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdIE4X7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.53.212:56067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 07:31:03,552 [I] DB_PATCHES have already been applied
I20260926 07:31:03.558811 22743 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/minicluster-data/ranger-admin:/tmp/dist-test-taskdIE4X7/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskdIE4X7/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskdIE4X7/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdIE4X7/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 07:31:03.558889 22743 mini_ranger.cc:193] Using host: 127.22.53.212
I20260926 07:31:03.562556 22743 mini_ranger.cc:227] Ranger admin URL: http://127.22.53.212:53303
Sep 26, 2026 7:31:03 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 7:31:03 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 7:31:04 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 7:31:04 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 7:31:04 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 7:31:04 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 7:31:04 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 07:31:04.732522 22743 mini_ranger.cc:160] Time spent starting Ranger: real 9.340s user 0.000s sys 0.011s
Sep 26, 2026 7:31:26 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 7:31:26 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 26, 2026 7:31:26 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 7:31:26 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 7:31:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 7:31:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 7:31:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 7:31:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 7:31:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 7:31:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 7:31:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 7:31:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 7:31:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 7:31:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 7:31:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 7:31:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 7:31:26 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 7:31:27 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 07:31:28.276522 22743 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.22.53.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.22.53.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.22.53.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/krb5kdc/rangerkms_127.22.53.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.22.53.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/krb5kdc/rangerkms_127.22.53.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.22.53.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.22.53.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.53.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/krb5kdc/HTTP_127.22.53.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.22.53.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/krb5kdc/HTTP_127.22.53.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 07:31:28 dist-test-slave-wdjc krb5kdc[1034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407888, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 07:31:28.370029 22743 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 07:31:28.381256 22743 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 07:31:29.334972 22743 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 07:31:29.335109 22743 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/minicluster-data/ranger-kms
I20260926 07:31:29.335237 22743 mini_ranger_kms.cc:227] Using postgres at 127.22.53.212:56067
2026-09-26 07:31:29,536 [I] DB FLAVOR :POSTGRES
2026-09-26 07:31:29,537 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 07:31:29,537 [I] Checking connection
2026-09-26 07:31:29,537 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdIE4X7/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.53.212:56067/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 07:31:29,811 [I] connection success
2026-09-26 07:31:29,811 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 07:31:29,811 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 07:31:29,812 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdIE4X7/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.53.212:56067/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 07:31:30,074 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 07:31:30,074 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 07:31:30,074 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 07:31:30,075 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdIE4X7/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.53.212:56067/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 07:31:30,411 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 07:31:30.934787 22743 mini_ranger_kms.cc:310] Created kms service
I20260926 07:31:31.155900 22743 mini_ranger_kms.cc:326] Created kudu user
I20260926 07:31:31.212844 22743 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 07:31:31 dist-test-slave-wdjc krb5kdc[1034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407888, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.22.53.212@KRBTEST.COM
I20260926 07:31:31.658042 22743 mini_ranger_kms.cc:384] Added ranger policy
I20260926 07:31:31.658205 22743 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/minicluster-data/ranger-kms:/tmp/dist-test-taskdIE4X7/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdIE4X7/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskdIE4X7/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 07:31:31.658229 22743 mini_ranger_kms.cc:240] Using host: 127.22.53.212
I20260926 07:31:31.662040 22743 mini_ranger_kms.cc:276] Ranger KMS PID: 1428
I20260926 07:31:31.662204 22743 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.22.53.212:53523
Sep 26, 2026 7:31:32 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 7:31:32 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 7:31:33 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 7:31:33 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 7:31:33 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 7:31:33 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260926 07:31:33.992697 22743 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.623s user 0.011s sys 0.175s
I20260926 07:31:33.992852 22743 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 07:31:33.992983 22743 mini_ranger_kms.cc:401] 127.22.53.212:53523/kms/v1/keys
Sep 26 07:31:34 dist-test-slave-wdjc krb5kdc[1034](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790407894, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.22.53.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 26 07:31:34 dist-test-slave-wdjc krb5kdc[1034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407894, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.22.53.212@KRBTEST.COM for HTTP/127.22.53.212@KRBTEST.COM
Sep 26 07:31:35 dist-test-slave-wdjc krb5kdc[1034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407894, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.22.53.212@KRBTEST.COM for HTTP/127.22.53.212@KRBTEST.COM
Sep 26 07:31:37 dist-test-slave-wdjc krb5kdc[1034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407897, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.22.53.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.53.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.53.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab.
Entry for principal kudu/127.22.53.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab.
I20260926 07:31:37.760720 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.53.254:42749
--webserver_interface=127.22.53.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab
--principal=kudu/127.22.53.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:45459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.53.254:42749
--ranger_config_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.53.212:53523/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:31:37.893977 1478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:31:37.894366 1478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:31:37.894446 1478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:31:37.898974 1478 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:31:37.899106 1478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:31:37.899142 1478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:31:37.899184 1478 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:31:37.899221 1478 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:31:37.905586 1478 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:45459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/minicluster-data/master-0/wal
--ranger_kms_url=127.22.53.212:53523/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.22.53.254:42749
--ranger_config_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab
--principal=kudu/127.22.53.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.53.254:42749
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.53.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1478
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:31:37.906924 1478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:31:37.908109 1478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:31:37.919737 1483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:31:37.919831 1478 server_base.cc:1034] running on GCE node
W20260926 07:31:37.919744 1484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:31:37.919744 1486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:31:37.920855 1478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:31:37.922266 1478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:31:37.923539 1478 hybrid_clock.cc:648] HybridClock initialized: now 1790407897923524 us; error 44 us; skew 500 ppm
Sep 26 07:31:37 dist-test-slave-wdjc krb5kdc[1034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407897, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:31:37.927203 1478 init.cc:377] Logged in from keytab as kudu/127.22.53.254@KRBTEST.COM (short username kudu)
I20260926 07:31:37.928686 1478 webserver.cc:460] Webserver started at http://127.22.53.254:41429/ using document root <none> and password file <none>
I20260926 07:31:37.929358 1478 fs_manager.cc:359] Metadata directory not provided
I20260926 07:31:37.929456 1478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:31:37.929701 1478 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 07:31:37 dist-test-slave-wdjc krb5kdc[1034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407897, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.254@KRBTEST.COM for HTTP/127.22.53.212@KRBTEST.COM
I20260926 07:31:37.986553 1478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/minicluster-data/master-0/data/instance:
uuid: "ef8f6d76ab144909a456b7f4ff4e2bc1"
format_stamp: "Formatted at 2026-09-26 07:31:37 on dist-test-slave-wdjc"
server_key: "7858aaa3e0e5b9dc75e5c6f9b4ac9da5"
server_key_iv: "267578e643336740f078cce409fcfe35"
server_key_version: "kuduclusterkey@0"
I20260926 07:31:37.987378 1478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/minicluster-data/master-0/wal/instance:
uuid: "ef8f6d76ab144909a456b7f4ff4e2bc1"
format_stamp: "Formatted at 2026-09-26 07:31:37 on dist-test-slave-wdjc"
server_key: "7858aaa3e0e5b9dc75e5c6f9b4ac9da5"
server_key_iv: "267578e643336740f078cce409fcfe35"
server_key_version: "kuduclusterkey@0"
I20260926 07:31:37.992138 1478 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260926 07:31:38.010681 1493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:31:38.012369 1478 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 07:31:38.012593 1478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/minicluster-data/master-0/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/minicluster-data/master-0/wal
uuid: "ef8f6d76ab144909a456b7f4ff4e2bc1"
format_stamp: "Formatted at 2026-09-26 07:31:37 on dist-test-slave-wdjc"
server_key: "7858aaa3e0e5b9dc75e5c6f9b4ac9da5"
server_key_iv: "267578e643336740f078cce409fcfe35"
server_key_version: "kuduclusterkey@0"
I20260926 07:31:38.012771 1478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:31:38.028084 1478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:31:38.032330 1478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:31:38.032691 1478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:31:38.041594 1478 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.254:42749
I20260926 07:31:38.041636 1545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.254:42749 every 8 connection(s)
I20260926 07:31:38.042765 1478 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
I20260926 07:31:38.047132 1546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:31:38.050892 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 1478
I20260926 07:31:38.051054 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/minicluster-data/master-0/wal/instance
I20260926 07:31:38.051348 22743 external_mini_cluster.cc:1428] Setting key 52728089cacf93f65fcfecd39e86b78f
I20260926 07:31:38.055701 1546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef8f6d76ab144909a456b7f4ff4e2bc1: Bootstrap starting.
I20260926 07:31:38.059082 1546 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef8f6d76ab144909a456b7f4ff4e2bc1: Neither blocks nor log segments found. Creating new log.
I20260926 07:31:38.060226 1546 log.cc:824] T 00000000000000000000000000000000 P ef8f6d76ab144909a456b7f4ff4e2bc1: Log is configured to *not* fsync() on all Append() calls
Sep 26 07:31:38 dist-test-slave-wdjc krb5kdc[1034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407897, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.53.254@KRBTEST.COM
I20260926 07:31:38.063088 1546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef8f6d76ab144909a456b7f4ff4e2bc1: No bootstrap required, opened a new log
I20260926 07:31:38.067521 1549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:31:38.053057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45942 (local address 127.22.53.254:42749)
0926 07:31:38.053556 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0926 07:31:38.053573 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 07:31:38.053611 (+ 38us) server_negotiation.cc:407] Connection header received
0926 07:31:38.054332 (+ 721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:31:38.054352 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:31:38.054734 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:31:38.055124 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:31:38.056009 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:31:38.057151 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:31:38.058094 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:31:38.058639 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:31:38.058833 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:31:38.061796 (+ 2963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:31:38.061803 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:31:38.061819 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:31:38.061848 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:31:38.064392 (+ 2544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:31:38.064948 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:31:38.064954 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:31:38.064960 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:31:38.065084 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:31:38.065400 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:31:38.065404 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:31:38.065405 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:31:38.065850 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:31:38.066067 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0926 07:31:38.066358 (+ 291us) server_negotiation.cc:299] Negotiation successful
0926 07:31:38.066719 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":172,"threads_started":1}
I20260926 07:31:38.067471 1546 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef8f6d76ab144909a456b7f4ff4e2bc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef8f6d76ab144909a456b7f4ff4e2bc1" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 42749 } }
I20260926 07:31:38.067821 1546 raft_consensus.cc:374] T 00000000000000000000000000000000 P ef8f6d76ab144909a456b7f4ff4e2bc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:31:38.067893 1546 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef8f6d76ab144909a456b7f4ff4e2bc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef8f6d76ab144909a456b7f4ff4e2bc1, State: Initialized, Role: FOLLOWER
I20260926 07:31:38.068427 1546 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef8f6d76ab144909a456b7f4ff4e2bc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef8f6d76ab144909a456b7f4ff4e2bc1" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 42749 } }
I20260926 07:31:38.068670 1546 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef8f6d76ab144909a456b7f4ff4e2bc1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:31:38.068770 1546 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef8f6d76ab144909a456b7f4ff4e2bc1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:31:38.068910 1546 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef8f6d76ab144909a456b7f4ff4e2bc1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:31:38.070402 1546 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef8f6d76ab144909a456b7f4ff4e2bc1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef8f6d76ab144909a456b7f4ff4e2bc1" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 42749 } }
I20260926 07:31:38.070834 1546 leader_election.cc:302] T 00000000000000000000000000000000 P ef8f6d76ab144909a456b7f4ff4e2bc1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef8f6d76ab144909a456b7f4ff4e2bc1; no voters:
I20260926 07:31:38.071291 1546 leader_election.cc:288] T 00000000000000000000000000000000 P ef8f6d76ab144909a456b7f4ff4e2bc1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:31:38.071432 1551 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef8f6d76ab144909a456b7f4ff4e2bc1 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:31:38.071735 1551 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef8f6d76ab144909a456b7f4ff4e2bc1 [term 1 LEADER]: Becoming Leader. State: Replica: ef8f6d76ab144909a456b7f4ff4e2bc1, State: Running, Role: LEADER
I20260926 07:31:38.072166 1551 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef8f6d76ab144909a456b7f4ff4e2bc1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef8f6d76ab144909a456b7f4ff4e2bc1" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 42749 } }
I20260926 07:31:38.072870 1546 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef8f6d76ab144909a456b7f4ff4e2bc1 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:31:38.073603 1551 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef8f6d76ab144909a456b7f4ff4e2bc1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ef8f6d76ab144909a456b7f4ff4e2bc1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef8f6d76ab144909a456b7f4ff4e2bc1" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 42749 } } }
I20260926 07:31:38.073843 1551 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef8f6d76ab144909a456b7f4ff4e2bc1 [sys.catalog]: This master's current role is: LEADER
I20260926 07:31:38.074106 1551 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef8f6d76ab144909a456b7f4ff4e2bc1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef8f6d76ab144909a456b7f4ff4e2bc1. Latest consensus state: current_term: 1 leader_uuid: "ef8f6d76ab144909a456b7f4ff4e2bc1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef8f6d76ab144909a456b7f4ff4e2bc1" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 42749 } } }
I20260926 07:31:38.074261 1551 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef8f6d76ab144909a456b7f4ff4e2bc1 [sys.catalog]: This master's current role is: LEADER
I20260926 07:31:38.074582 1559 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:31:38.078526 1559 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:31:38.079658 1546 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 07:31:38.089337 1559 catalog_manager.cc:1347] Generated new cluster ID: b1aaa160c7404aac99dc86672ea128d8
I20260926 07:31:38.089533 1559 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:31:38.100432 1559 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:31:38.102092 1559 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:31:38.121946 1559 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ef8f6d76ab144909a456b7f4ff4e2bc1: Generated new TSK 0
I20260926 07:31:38.123113 1559 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.53.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.53.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.53.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.193.keytab.
Entry for principal kudu/127.22.53.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.193.keytab.
I20260926 07:31:39.020411 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.53.212:53523/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.53.193:0
--local_ip_for_outbound_sockets=127.22.53.193
--webserver_interface=127.22.53.193
--webserver_port=0
--tserver_master_addrs=127.22.53.254:42749
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.193.keytab
--principal=kudu/127.22.53.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:45459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:31:39.162911 1591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:31:39.163439 1591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:31:39.163570 1591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:31:39.170306 1591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:31:39.170519 1591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:31:39.170677 1591 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.53.193
I20260926 07:31:39.178126 1591 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:45459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.22.53.212:53523/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.193.keytab
--principal=kudu/127.22.53.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.53.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.53.193
--webserver_port=0
--tserver_master_addrs=127.22.53.254:42749
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1591
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.53.193
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:31:39.181286 1591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:31:39.182852 1591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:31:39.200812 1601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:31:39.200975 1599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:31:39.200816 1598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:31:39.201481 1591 server_base.cc:1034] running on GCE node
I20260926 07:31:39.202296 1591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:31:39.203357 1591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:31:39.204634 1591 hybrid_clock.cc:648] HybridClock initialized: now 1790407899204603 us; error 52 us; skew 500 ppm
Sep 26 07:31:39 dist-test-slave-wdjc krb5kdc[1034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407899, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:31:39.209118 1591 init.cc:377] Logged in from keytab as kudu/127.22.53.193@KRBTEST.COM (short username kudu)
I20260926 07:31:39.210450 1591 webserver.cc:460] Webserver started at http://127.22.53.193:41133/ using document root <none> and password file <none>
I20260926 07:31:39.211251 1591 fs_manager.cc:359] Metadata directory not provided
I20260926 07:31:39.211344 1591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:31:39.211592 1591 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 07:31:39 dist-test-slave-wdjc krb5kdc[1034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407899, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.193@KRBTEST.COM for HTTP/127.22.53.212@KRBTEST.COM
I20260926 07:31:39.231546 1591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/minicluster-data/ts-0/data/instance:
uuid: "b0f6094c72904fae960314ff0e9a7ba3"
format_stamp: "Formatted at 2026-09-26 07:31:39 on dist-test-slave-wdjc"
server_key: "d8551aa1cb3c57f34af861bc132b68b3"
server_key_iv: "0b1c140a3af65a2f72e3c22105f6aa9d"
server_key_version: "kuduclusterkey@0"
I20260926 07:31:39.232439 1591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/minicluster-data/ts-0/wal/instance:
uuid: "b0f6094c72904fae960314ff0e9a7ba3"
format_stamp: "Formatted at 2026-09-26 07:31:39 on dist-test-slave-wdjc"
server_key: "d8551aa1cb3c57f34af861bc132b68b3"
server_key_iv: "0b1c140a3af65a2f72e3c22105f6aa9d"
server_key_version: "kuduclusterkey@0"
I20260926 07:31:39.239236 1591 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260926 07:31:39.273685 1608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:31:39.276053 1591 fs_manager.cc:725] Time spent opening block manager: real 0.021s user 0.004s sys 0.000s
I20260926 07:31:39.276283 1591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/minicluster-data/ts-0/wal
uuid: "b0f6094c72904fae960314ff0e9a7ba3"
format_stamp: "Formatted at 2026-09-26 07:31:39 on dist-test-slave-wdjc"
server_key: "d8551aa1cb3c57f34af861bc132b68b3"
server_key_iv: "0b1c140a3af65a2f72e3c22105f6aa9d"
server_key_version: "kuduclusterkey@0"
I20260926 07:31:39.276480 1591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:31:39.300364 1591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:31:39.317428 1591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:31:39.317692 1591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:31:39.318589 1591 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:31:39.319818 1591 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:31:39.319913 1591 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:31:39.319988 1591 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:31:39.320025 1591 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:31:39.332839 1591 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.193:37849
I20260926 07:31:39.334249 1591 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/minicluster-data/ts-0/data/info.pb
I20260926 07:31:39.335266 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 1591
I20260926 07:31:39.335441 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/minicluster-data/ts-0/wal/instance
I20260926 07:31:39.335783 22743 external_mini_cluster.cc:1428] Setting key f27f308be1167dd960d24b9639014299
I20260926 07:31:39.342147 1723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.193:37849 every 8 connection(s)
WARNING: no policy specified for kudu/127.22.53.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.53.194@KRBTEST.COM" created.
Sep 26 07:31:39 dist-test-slave-wdjc krb5kdc[1034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407899, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.193@KRBTEST.COM for kudu/127.22.53.254@KRBTEST.COM
I20260926 07:31:39.374094 1727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:31:39.340858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.193:50155 (local address 127.22.53.254:42749)
0926 07:31:39.352803 (+ 11945us) server_negotiation.cc:206] Beginning negotiation
0926 07:31:39.352809 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:31:39.352851 (+ 42us) server_negotiation.cc:407] Connection header received
0926 07:31:39.353717 (+ 866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:31:39.353723 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:31:39.353797 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:31:39.353871 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:31:39.358799 (+ 4928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:31:39.359785 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:31:39.362058 (+ 2273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:31:39.362467 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:31:39.362572 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:31:39.366550 (+ 3978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:31:39.366555 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:31:39.366558 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:31:39.366593 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:31:39.368884 (+ 2291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:31:39.371677 (+ 2793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:31:39.371682 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:31:39.371684 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:31:39.371817 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:31:39.372361 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:31:39.372365 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:31:39.372367 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:31:39.372734 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:31:39.372835 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 07:31:39.373711 (+ 876us) server_negotiation.cc:299] Negotiation successful
0926 07:31:39.373884 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11806,"thread_start_us":109,"threads_started":1}
I20260926 07:31:39.376482 1728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:31:39.342015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:42749 (local address 127.22.53.193:50155)
0926 07:31:39.351253 (+ 9238us) negotiation.cc:107] Waiting for socket to connect
0926 07:31:39.351384 (+ 131us) client_negotiation.cc:174] Beginning negotiation
0926 07:31:39.353293 (+ 1909us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:31:39.354625 (+ 1332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:31:39.354638 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:31:39.355069 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:31:39.358512 (+ 3443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:31:39.358549 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:31:39.360022 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:31:39.360027 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:31:39.361817 (+ 1790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:31:39.361836 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:31:39.362955 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:31:39.362959 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:31:39.363208 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:31:39.364165 (+ 957us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:31:39.364191 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:31:39.366251 (+ 2060us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:31:39.369088 (+ 2837us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:31:39.369096 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:31:39.369119 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:31:39.369573 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:31:39.371959 (+ 2386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:31:39.371962 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:31:39.371965 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:31:39.372217 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:31:39.372917 (+ 700us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:31:39.372925 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:31:39.373345 (+ 420us) client_negotiation.cc:769] Sending connection context
0926 07:31:39.374786 (+ 1441us) client_negotiation.cc:240] Negotiation successful
0926 07:31:39.375240 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8864,"thread_start_us":194,"threads_started":1}
I20260926 07:31:39.383196 1724 heartbeater.cc:347] Connected to a master server at 127.22.53.254:42749
I20260926 07:31:39.383561 1724 heartbeater.cc:464] Registering TS with master...
I20260926 07:31:39.384290 1724 heartbeater.cc:511] Master 127.22.53.254:42749 requested a full tablet report, sending...
I20260926 07:31:39.386853 1510 ts_manager.cc:194] Registered new tserver with Master: b0f6094c72904fae960314ff0e9a7ba3 (127.22.53.193:37849)
I20260926 07:31:39.388907 1510 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.53.193@KRBTEST.COM'} at 127.22.53.193:50155
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.53.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.194.keytab.
Entry for principal kudu/127.22.53.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.194.keytab.
I20260926 07:31:39.400898 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.53.212:53523/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.53.194:0
--local_ip_for_outbound_sockets=127.22.53.194
--webserver_interface=127.22.53.194
--webserver_port=0
--tserver_master_addrs=127.22.53.254:42749
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.194.keytab
--principal=kudu/127.22.53.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:45459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 07:31:39 dist-test-slave-wdjc krb5kdc[1034](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790407899, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260926 07:31:39.594406 1730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:31:39.594870 1730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:31:39.594956 1730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:31:39.601163 1730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:31:39.601269 1730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:31:39.601358 1730 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.53.194
I20260926 07:31:39.609746 1730 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:45459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.22.53.212:53523/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.194.keytab
--principal=kudu/127.22.53.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.53.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.53.194
--webserver_port=0
--tserver_master_addrs=127.22.53.254:42749
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1730
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.53.194
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:31:39.611299 1730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:31:39.612757 1730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:31:39.627877 1736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:31:39.628369 1730 server_base.cc:1034] running on GCE node
W20260926 07:31:39.628655 1737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:31:39.628945 1739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:31:39.629530 1730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:31:39.630362 1730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:31:39.631543 1730 hybrid_clock.cc:648] HybridClock initialized: now 1790407899631478 us; error 88 us; skew 500 ppm
Sep 26 07:31:39 dist-test-slave-wdjc krb5kdc[1034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407899, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:31:39.635378 1730 init.cc:377] Logged in from keytab as kudu/127.22.53.194@KRBTEST.COM (short username kudu)
I20260926 07:31:39.636680 1730 webserver.cc:460] Webserver started at http://127.22.53.194:41291/ using document root <none> and password file <none>
I20260926 07:31:39.637725 1730 fs_manager.cc:359] Metadata directory not provided
I20260926 07:31:39.638213 1730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:31:39.638531 1730 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 07:31:39 dist-test-slave-wdjc krb5kdc[1034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407899, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.194@KRBTEST.COM for HTTP/127.22.53.212@KRBTEST.COM
I20260926 07:31:39.657362 1730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/minicluster-data/ts-1/data/instance:
uuid: "9e2f7f60f1324e719677220539a0858b"
format_stamp: "Formatted at 2026-09-26 07:31:39 on dist-test-slave-wdjc"
server_key: "513bc5b515c5214e7ec7becd59b3215f"
server_key_iv: "6d995a9362ab64c0bb0464a1e8159788"
server_key_version: "kuduclusterkey@0"
I20260926 07:31:39.660509 1730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/minicluster-data/ts-1/wal/instance:
uuid: "9e2f7f60f1324e719677220539a0858b"
format_stamp: "Formatted at 2026-09-26 07:31:39 on dist-test-slave-wdjc"
server_key: "513bc5b515c5214e7ec7becd59b3215f"
server_key_iv: "6d995a9362ab64c0bb0464a1e8159788"
server_key_version: "kuduclusterkey@0"
I20260926 07:31:39.667213 1730 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.003s
I20260926 07:31:39.694792 1746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:31:39.696564 1730 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20260926 07:31:39.696724 1730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/minicluster-data/ts-1/wal
uuid: "9e2f7f60f1324e719677220539a0858b"
format_stamp: "Formatted at 2026-09-26 07:31:39 on dist-test-slave-wdjc"
server_key: "513bc5b515c5214e7ec7becd59b3215f"
server_key_iv: "6d995a9362ab64c0bb0464a1e8159788"
server_key_version: "kuduclusterkey@0"
I20260926 07:31:39.696868 1730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:31:39.763535 1730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:31:39.772039 1730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:31:39.772360 1730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:31:39.773269 1730 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:31:39.774564 1730 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:31:39.774677 1730 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:31:39.774773 1730 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:31:39.774833 1730 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:31:39.789889 1730 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.194:33065
I20260926 07:31:39.791364 1730 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/minicluster-data/ts-1/data/info.pb
I20260926 07:31:39.797891 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 1730
I20260926 07:31:39.798049 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/minicluster-data/ts-1/wal/instance
I20260926 07:31:39.798394 22743 external_mini_cluster.cc:1428] Setting key 7b11ef9f3fef0b6454ed94e773990b75
I20260926 07:31:39.792927 1859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.194:33065 every 8 connection(s)
Sep 26 07:31:39 dist-test-slave-wdjc krb5kdc[1034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407899, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.194@KRBTEST.COM for kudu/127.22.53.254@KRBTEST.COM
I20260926 07:31:39.818902 1727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:31:39.797527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.194:34537 (local address 127.22.53.254:42749)
0926 07:31:39.797712 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 07:31:39.797718 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:31:39.799640 (+ 1922us) server_negotiation.cc:407] Connection header received
0926 07:31:39.803713 (+ 4073us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:31:39.803718 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:31:39.803791 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:31:39.803848 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:31:39.805950 (+ 2102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:31:39.806777 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:31:39.808789 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:31:39.809151 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:31:39.809237 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:31:39.813936 (+ 4699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:31:39.813941 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:31:39.813945 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:31:39.813981 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:31:39.815880 (+ 1899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:31:39.816821 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:31:39.816825 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:31:39.816827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:31:39.816927 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:31:39.817394 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:31:39.817398 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:31:39.817400 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:31:39.817653 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:31:39.817752 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 07:31:39.818564 (+ 812us) server_negotiation.cc:299] Negotiation successful
0926 07:31:39.818721 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 07:31:39.821126 1862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:31:39.796141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:42749 (local address 127.22.53.194:34537)
0926 07:31:39.799434 (+ 3293us) negotiation.cc:107] Waiting for socket to connect
0926 07:31:39.799520 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0926 07:31:39.803282 (+ 3762us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:31:39.804397 (+ 1115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:31:39.804411 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:31:39.804858 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:31:39.805695 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:31:39.805721 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:31:39.807006 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:31:39.807009 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:31:39.808605 (+ 1596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:31:39.808621 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:31:39.809501 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:31:39.809504 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:31:39.809738 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:31:39.811678 (+ 1940us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:31:39.811705 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:31:39.813630 (+ 1925us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:31:39.816072 (+ 2442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:31:39.816080 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:31:39.816105 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:31:39.816688 (+ 583us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:31:39.817043 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:31:39.817054 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:31:39.817056 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:31:39.817269 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:31:39.817868 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:31:39.817876 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:31:39.818242 (+ 366us) client_negotiation.cc:769] Sending connection context
0926 07:31:39.819527 (+ 1285us) client_negotiation.cc:240] Negotiation successful
0926 07:31:39.819948 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2946,"thread_start_us":185,"threads_started":1}
I20260926 07:31:39.823205 1860 heartbeater.cc:347] Connected to a master server at 127.22.53.254:42749
I20260926 07:31:39.823494 1860 heartbeater.cc:464] Registering TS with master...
I20260926 07:31:39.824147 1860 heartbeater.cc:511] Master 127.22.53.254:42749 requested a full tablet report, sending...
I20260926 07:31:39.825755 1510 ts_manager.cc:194] Registered new tserver with Master: 9e2f7f60f1324e719677220539a0858b (127.22.53.194:33065)
I20260926 07:31:39.826565 1510 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.53.194@KRBTEST.COM'} at 127.22.53.194:34537
WARNING: no policy specified for kudu/127.22.53.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.53.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.53.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.195.keytab.
Entry for principal kudu/127.22.53.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.195.keytab.
I20260926 07:31:39.867847 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.53.212:53523/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.53.195:0
--local_ip_for_outbound_sockets=127.22.53.195
--webserver_interface=127.22.53.195
--webserver_port=0
--tserver_master_addrs=127.22.53.254:42749
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.195.keytab
--principal=kudu/127.22.53.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:45459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 07:31:39 dist-test-slave-wdjc krb5kdc[1034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407899, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.254@KRBTEST.COM for HTTP/127.22.53.212@KRBTEST.COM
W20260926 07:31:40.066354 1865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:31:40.067392 1865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:31:40.067564 1865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:31:40.074205 1865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:31:40.074471 1865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:31:40.074620 1865 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.53.195
I20260926 07:31:40.083540 1865 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:45459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.22.53.212:53523/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.195.keytab
--principal=kudu/127.22.53.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.53.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.53.195
--webserver_port=0
--tserver_master_addrs=127.22.53.254:42749
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1865
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.53.195
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:31:40.085364 1865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:31:40.086962 1865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:31:40.106137 1870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:31:40.106097 1871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:31:40.106673 1873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:31:40.106838 1865 server_base.cc:1034] running on GCE node
I20260926 07:31:40.107448 1865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:31:40.108346 1865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:31:40.111650 1865 hybrid_clock.cc:648] HybridClock initialized: now 1790407900111559 us; error 99 us; skew 500 ppm
Sep 26 07:31:40 dist-test-slave-wdjc krb5kdc[1034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407900, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:31:40.115939 1865 init.cc:377] Logged in from keytab as kudu/127.22.53.195@KRBTEST.COM (short username kudu)
I20260926 07:31:40.117324 1865 webserver.cc:460] Webserver started at http://127.22.53.195:44589/ using document root <none> and password file <none>
I20260926 07:31:40.118124 1865 fs_manager.cc:359] Metadata directory not provided
I20260926 07:31:40.118239 1865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:31:40.118502 1865 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 07:31:40 dist-test-slave-wdjc krb5kdc[1034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407900, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.195@KRBTEST.COM for HTTP/127.22.53.212@KRBTEST.COM
I20260926 07:31:40.165413 1865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/minicluster-data/ts-2/data/instance:
uuid: "04923a7e4332423296ab247949e8d9b0"
format_stamp: "Formatted at 2026-09-26 07:31:40 on dist-test-slave-wdjc"
server_key: "5af74680e4755b9e1cd8355f9c39b685"
server_key_iv: "bf076c94456b2f9693adb649cf1b51b9"
server_key_version: "kuduclusterkey@0"
I20260926 07:31:40.166359 1865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/minicluster-data/ts-2/wal/instance:
uuid: "04923a7e4332423296ab247949e8d9b0"
format_stamp: "Formatted at 2026-09-26 07:31:40 on dist-test-slave-wdjc"
server_key: "5af74680e4755b9e1cd8355f9c39b685"
server_key_iv: "bf076c94456b2f9693adb649cf1b51b9"
server_key_version: "kuduclusterkey@0"
I20260926 07:31:40.172614 1865 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260926 07:31:40.216984 1880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:31:40.218971 1865 fs_manager.cc:725] Time spent opening block manager: real 0.018s user 0.002s sys 0.000s
I20260926 07:31:40.219142 1865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/minicluster-data/ts-2/wal
uuid: "04923a7e4332423296ab247949e8d9b0"
format_stamp: "Formatted at 2026-09-26 07:31:40 on dist-test-slave-wdjc"
server_key: "5af74680e4755b9e1cd8355f9c39b685"
server_key_iv: "bf076c94456b2f9693adb649cf1b51b9"
server_key_version: "kuduclusterkey@0"
I20260926 07:31:40.219302 1865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:31:40.275200 1865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:31:40.282671 1865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:31:40.283207 1865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:31:40.284982 1865 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:31:40.286258 1865 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:31:40.286342 1865 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:31:40.286401 1865 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:31:40.286427 1865 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:31:40.298183 1865 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.195:33713
I20260926 07:31:40.299371 1865 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/minicluster-data/ts-2/data/info.pb
I20260926 07:31:40.300264 1993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.195:33713 every 8 connection(s)
I20260926 07:31:40.300916 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 1865
I20260926 07:31:40.301043 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/minicluster-data/ts-2/wal/instance
I20260926 07:31:40.301349 22743 external_mini_cluster.cc:1428] Setting key 70dd6caace5f71b436f21f75b6139caf
Sep 26 07:31:40 dist-test-slave-wdjc krb5kdc[1034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407900, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.195@KRBTEST.COM for kudu/127.22.53.254@KRBTEST.COM
I20260926 07:31:40.324162 1727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:31:40.306165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.195:53937 (local address 127.22.53.254:42749)
0926 07:31:40.306301 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0926 07:31:40.306308 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:31:40.306878 (+ 570us) server_negotiation.cc:407] Connection header received
0926 07:31:40.308559 (+ 1681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:31:40.308564 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:31:40.308630 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:31:40.308693 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:31:40.310689 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:31:40.311561 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:31:40.313611 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:31:40.314035 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:31:40.314140 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:31:40.318666 (+ 4526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:31:40.318671 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:31:40.318674 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:31:40.318710 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:31:40.320978 (+ 2268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:31:40.321876 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:31:40.321881 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:31:40.321883 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:31:40.322044 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:31:40.322582 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:31:40.322585 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:31:40.322587 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:31:40.322875 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:31:40.322968 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 07:31:40.323796 (+ 828us) server_negotiation.cc:299] Negotiation successful
0926 07:31:40.323983 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 07:31:40.326663 1996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:31:40.304860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:42749 (local address 127.22.53.195:53937)
0926 07:31:40.306687 (+ 1827us) negotiation.cc:107] Waiting for socket to connect
0926 07:31:40.306754 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0926 07:31:40.307953 (+ 1199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:31:40.309227 (+ 1274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:31:40.309238 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:31:40.309676 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:31:40.310471 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:31:40.310495 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:31:40.311757 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:31:40.311760 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:31:40.313418 (+ 1658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:31:40.313436 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:31:40.314408 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:31:40.314416 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:31:40.314664 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:31:40.315552 (+ 888us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:31:40.315577 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:31:40.318411 (+ 2834us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:31:40.321197 (+ 2786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:31:40.321205 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:31:40.321229 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:31:40.321715 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:31:40.322211 (+ 496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:31:40.322214 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:31:40.322217 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:31:40.322449 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:31:40.323047 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:31:40.323055 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:31:40.323458 (+ 403us) client_negotiation.cc:769] Sending connection context
0926 07:31:40.324749 (+ 1291us) client_negotiation.cc:240] Negotiation successful
0926 07:31:40.325244 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1588,"thread_start_us":150,"threads_started":1}
I20260926 07:31:40.328756 1994 heartbeater.cc:347] Connected to a master server at 127.22.53.254:42749
I20260926 07:31:40.329073 1994 heartbeater.cc:464] Registering TS with master...
I20260926 07:31:40.329871 1994 heartbeater.cc:511] Master 127.22.53.254:42749 requested a full tablet report, sending...
I20260926 07:31:40.331624 1509 ts_manager.cc:194] Registered new tserver with Master: 04923a7e4332423296ab247949e8d9b0 (127.22.53.195:33713)
I20260926 07:31:40.332441 1509 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.53.195@KRBTEST.COM'} at 127.22.53.195:53937
I20260926 07:31:40.332667 22743 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:31:40.347685 1727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:31:40.335700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45956 (local address 127.22.53.254:42749)
0926 07:31:40.335845 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 07:31:40.335850 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:31:40.336563 (+ 713us) server_negotiation.cc:407] Connection header received
0926 07:31:40.336828 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:31:40.336832 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:31:40.336893 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:31:40.336963 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:31:40.338174 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:31:40.339019 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:31:40.340312 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:31:40.340674 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:31:40.340768 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:31:40.343275 (+ 2507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:31:40.343281 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:31:40.343284 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:31:40.343325 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:31:40.345349 (+ 2024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:31:40.346027 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:31:40.346034 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:31:40.346036 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:31:40.346147 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:31:40.346521 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:31:40.346525 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:31:40.346527 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:31:40.346792 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:31:40.346929 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0926 07:31:40.347362 (+ 433us) server_negotiation.cc:299] Negotiation successful
0926 07:31:40.347512 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 07:31:40.352272 1509 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45956:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:31:40.357359 1509 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
Sep 26 07:31:40 dist-test-slave-wdjc krb5kdc[1034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407899, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.254@KRBTEST.COM for HTTP/127.22.53.212@KRBTEST.COM
I20260926 07:31:40.390944 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:31:40.371501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.194:33065 (local address 127.0.0.1:60464)
0926 07:31:40.379364 (+ 7863us) negotiation.cc:107] Waiting for socket to connect
0926 07:31:40.379442 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0926 07:31:40.379582 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:31:40.380880 (+ 1298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:31:40.380887 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:31:40.380912 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:31:40.381057 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:31:40.381063 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:31:40.386440 (+ 5377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:31:40.386444 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:31:40.387572 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:31:40.387585 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:31:40.390328 (+ 2743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:31:40.390332 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:31:40.390470 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:31:40.390509 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 07:31:40.390628 (+ 119us) client_negotiation.cc:240] Negotiation successful
0926 07:31:40.390772 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7634,"thread_start_us":105,"threads_started":1}
I20260926 07:31:40.391782 2007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:31:40.373322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.193:37849 (local address 127.0.0.1:52494)
0926 07:31:40.379807 (+ 6485us) negotiation.cc:107] Waiting for socket to connect
0926 07:31:40.379819 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 07:31:40.379891 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:31:40.381755 (+ 1864us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:31:40.381758 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:31:40.381777 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:31:40.381884 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:31:40.381890 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:31:40.384524 (+ 2634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:31:40.384549 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:31:40.385976 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:31:40.385992 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:31:40.391384 (+ 5392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:31:40.391388 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:31:40.391490 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:31:40.391515 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 07:31:40.391585 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 07:31:40.391654 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6426,"thread_start_us":127,"threads_started":1}
I20260926 07:31:40.392395 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:31:40.372376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60464 (local address 127.22.53.194:33065)
0926 07:31:40.380130 (+ 7754us) server_negotiation.cc:206] Beginning negotiation
0926 07:31:40.380138 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:31:40.380163 (+ 25us) server_negotiation.cc:407] Connection header received
0926 07:31:40.380244 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:31:40.380249 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:31:40.380506 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:31:40.380686 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:31:40.383110 (+ 2424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:31:40.386310 (+ 3200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:31:40.389117 (+ 2807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:31:40.390000 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:31:40.390110 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:31:40.390250 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 07:31:40.392097 (+ 1847us) server_negotiation.cc:299] Negotiation successful
0926 07:31:40.392233 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7642,"thread_start_us":81,"threads_started":1}
I20260926 07:31:40.392975 1724 heartbeater.cc:503] Master 127.22.53.254:42749 was elected leader, sending a full tablet report...
I20260926 07:31:40.394001 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:31:40.379233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.195:33713 (local address 127.0.0.1:57152)
0926 07:31:40.380781 (+ 1548us) negotiation.cc:107] Waiting for socket to connect
0926 07:31:40.380796 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:31:40.380896 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:31:40.388448 (+ 7552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:31:40.388453 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:31:40.388489 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:31:40.388661 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:31:40.388668 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:31:40.390179 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:31:40.390182 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:31:40.391619 (+ 1437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:31:40.391635 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:31:40.393535 (+ 1900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:31:40.393539 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:31:40.393642 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:31:40.393672 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 07:31:40.393750 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 07:31:40.393852 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1486,"thread_start_us":134,"threads_started":1}
I20260926 07:31:40.394454 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:31:40.379741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57152 (local address 127.22.53.195:33713)
0926 07:31:40.380631 (+ 890us) server_negotiation.cc:206] Beginning negotiation
0926 07:31:40.380643 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 07:31:40.380991 (+ 348us) server_negotiation.cc:407] Connection header received
0926 07:31:40.381072 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:31:40.381077 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:31:40.384152 (+ 3075us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:31:40.384345 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:31:40.388838 (+ 4493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:31:40.390018 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:31:40.392082 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:31:40.393176 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:31:40.393299 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:31:40.393453 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 07:31:40.394188 (+ 735us) server_negotiation.cc:299] Negotiation successful
0926 07:31:40.394309 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":762,"thread_start_us":95,"threads_started":1}
I20260926 07:31:40.396678 1794 tablet_service.cc:1467] Processing CreateTablet for tablet c38056c2241b498196e240cb04439711 (DEFAULT_TABLE table=test-table [id=1b492a0df8eb4d35bb7fb997c66b141a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:31:40.397400 1920 tablet_service.cc:1467] Processing CreateTablet for tablet c38056c2241b498196e240cb04439711 (DEFAULT_TABLE table=test-table [id=1b492a0df8eb4d35bb7fb997c66b141a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:31:40.398005 1794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c38056c2241b498196e240cb04439711. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:31:40.398447 1920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c38056c2241b498196e240cb04439711. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:31:40.406782 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:31:40.374674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52494 (local address 127.22.53.193:37849)
0926 07:31:40.379014 (+ 4340us) server_negotiation.cc:206] Beginning negotiation
0926 07:31:40.379027 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 07:31:40.381187 (+ 2160us) server_negotiation.cc:407] Connection header received
0926 07:31:40.381270 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:31:40.381275 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:31:40.381460 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:31:40.381590 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:31:40.382036 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:31:40.384380 (+ 2344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:31:40.387723 (+ 3343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:31:40.388704 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:31:40.388821 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:31:40.389001 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0926 07:31:40.406462 (+ 17461us) server_negotiation.cc:299] Negotiation successful
0926 07:31:40.406623 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4189,"thread_start_us":106,"threads_started":1}
I20260926 07:31:40.411425 1652 tablet_service.cc:1467] Processing CreateTablet for tablet c38056c2241b498196e240cb04439711 (DEFAULT_TABLE table=test-table [id=1b492a0df8eb4d35bb7fb997c66b141a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:31:40.413033 1652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c38056c2241b498196e240cb04439711. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:31:40.422101 2013 tablet_bootstrap.cc:492] T c38056c2241b498196e240cb04439711 P 04923a7e4332423296ab247949e8d9b0: Bootstrap starting.
I20260926 07:31:40.422670 2014 tablet_bootstrap.cc:492] T c38056c2241b498196e240cb04439711 P 9e2f7f60f1324e719677220539a0858b: Bootstrap starting.
I20260926 07:31:40.426788 2013 tablet_bootstrap.cc:654] T c38056c2241b498196e240cb04439711 P 04923a7e4332423296ab247949e8d9b0: Neither blocks nor log segments found. Creating new log.
I20260926 07:31:40.426775 2014 tablet_bootstrap.cc:654] T c38056c2241b498196e240cb04439711 P 9e2f7f60f1324e719677220539a0858b: Neither blocks nor log segments found. Creating new log.
I20260926 07:31:40.427829 2013 log.cc:824] T c38056c2241b498196e240cb04439711 P 04923a7e4332423296ab247949e8d9b0: Log is configured to *not* fsync() on all Append() calls
I20260926 07:31:40.427940 2014 log.cc:824] T c38056c2241b498196e240cb04439711 P 9e2f7f60f1324e719677220539a0858b: Log is configured to *not* fsync() on all Append() calls
I20260926 07:31:40.430488 2013 tablet_bootstrap.cc:492] T c38056c2241b498196e240cb04439711 P 04923a7e4332423296ab247949e8d9b0: No bootstrap required, opened a new log
I20260926 07:31:40.430682 2014 tablet_bootstrap.cc:492] T c38056c2241b498196e240cb04439711 P 9e2f7f60f1324e719677220539a0858b: No bootstrap required, opened a new log
I20260926 07:31:40.430866 2013 ts_tablet_manager.cc:1397] T c38056c2241b498196e240cb04439711 P 04923a7e4332423296ab247949e8d9b0: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260926 07:31:40.430914 2014 ts_tablet_manager.cc:1397] T c38056c2241b498196e240cb04439711 P 9e2f7f60f1324e719677220539a0858b: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260926 07:31:40.435158 2013 raft_consensus.cc:348] T c38056c2241b498196e240cb04439711 P 04923a7e4332423296ab247949e8d9b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e2f7f60f1324e719677220539a0858b" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 33065 } } peers { permanent_uuid: "b0f6094c72904fae960314ff0e9a7ba3" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 37849 } } peers { permanent_uuid: "04923a7e4332423296ab247949e8d9b0" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 33713 } }
I20260926 07:31:40.428910 2015 tablet_bootstrap.cc:492] T c38056c2241b498196e240cb04439711 P b0f6094c72904fae960314ff0e9a7ba3: Bootstrap starting.
I20260926 07:31:40.435668 2014 raft_consensus.cc:348] T c38056c2241b498196e240cb04439711 P 9e2f7f60f1324e719677220539a0858b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e2f7f60f1324e719677220539a0858b" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 33065 } } peers { permanent_uuid: "b0f6094c72904fae960314ff0e9a7ba3" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 37849 } } peers { permanent_uuid: "04923a7e4332423296ab247949e8d9b0" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 33713 } }
I20260926 07:31:40.435957 2014 raft_consensus.cc:374] T c38056c2241b498196e240cb04439711 P 9e2f7f60f1324e719677220539a0858b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:31:40.436028 2014 raft_consensus.cc:729] T c38056c2241b498196e240cb04439711 P 9e2f7f60f1324e719677220539a0858b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e2f7f60f1324e719677220539a0858b, State: Initialized, Role: FOLLOWER
I20260926 07:31:40.436578 2014 consensus_queue.cc:260] T c38056c2241b498196e240cb04439711 P 9e2f7f60f1324e719677220539a0858b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e2f7f60f1324e719677220539a0858b" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 33065 } } peers { permanent_uuid: "b0f6094c72904fae960314ff0e9a7ba3" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 37849 } } peers { permanent_uuid: "04923a7e4332423296ab247949e8d9b0" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 33713 } }
I20260926 07:31:40.437860 2014 ts_tablet_manager.cc:1428] T c38056c2241b498196e240cb04439711 P 9e2f7f60f1324e719677220539a0858b: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260926 07:31:40.438021 2013 raft_consensus.cc:374] T c38056c2241b498196e240cb04439711 P 04923a7e4332423296ab247949e8d9b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:31:40.438148 2013 raft_consensus.cc:729] T c38056c2241b498196e240cb04439711 P 04923a7e4332423296ab247949e8d9b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04923a7e4332423296ab247949e8d9b0, State: Initialized, Role: FOLLOWER
I20260926 07:31:40.438264 1860 heartbeater.cc:503] Master 127.22.53.254:42749 was elected leader, sending a full tablet report...
I20260926 07:31:40.438807 2013 consensus_queue.cc:260] T c38056c2241b498196e240cb04439711 P 04923a7e4332423296ab247949e8d9b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e2f7f60f1324e719677220539a0858b" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 33065 } } peers { permanent_uuid: "b0f6094c72904fae960314ff0e9a7ba3" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 37849 } } peers { permanent_uuid: "04923a7e4332423296ab247949e8d9b0" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 33713 } }
I20260926 07:31:40.439750 2015 tablet_bootstrap.cc:654] T c38056c2241b498196e240cb04439711 P b0f6094c72904fae960314ff0e9a7ba3: Neither blocks nor log segments found. Creating new log.
I20260926 07:31:40.440901 2013 ts_tablet_manager.cc:1428] T c38056c2241b498196e240cb04439711 P 04923a7e4332423296ab247949e8d9b0: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20260926 07:31:40.441946 2015 log.cc:824] T c38056c2241b498196e240cb04439711 P b0f6094c72904fae960314ff0e9a7ba3: Log is configured to *not* fsync() on all Append() calls
I20260926 07:31:40.442447 1994 heartbeater.cc:503] Master 127.22.53.254:42749 was elected leader, sending a full tablet report...
I20260926 07:31:40.446019 2015 tablet_bootstrap.cc:492] T c38056c2241b498196e240cb04439711 P b0f6094c72904fae960314ff0e9a7ba3: No bootstrap required, opened a new log
I20260926 07:31:40.446273 2015 ts_tablet_manager.cc:1397] T c38056c2241b498196e240cb04439711 P b0f6094c72904fae960314ff0e9a7ba3: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.000s
I20260926 07:31:40.449159 2019 raft_consensus.cc:482] T c38056c2241b498196e240cb04439711 P 04923a7e4332423296ab247949e8d9b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:31:40.449374 2019 raft_consensus.cc:504] T c38056c2241b498196e240cb04439711 P 04923a7e4332423296ab247949e8d9b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e2f7f60f1324e719677220539a0858b" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 33065 } } peers { permanent_uuid: "b0f6094c72904fae960314ff0e9a7ba3" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 37849 } } peers { permanent_uuid: "04923a7e4332423296ab247949e8d9b0" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 33713 } }
I20260926 07:31:40.450414 2015 raft_consensus.cc:348] T c38056c2241b498196e240cb04439711 P b0f6094c72904fae960314ff0e9a7ba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e2f7f60f1324e719677220539a0858b" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 33065 } } peers { permanent_uuid: "b0f6094c72904fae960314ff0e9a7ba3" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 37849 } } peers { permanent_uuid: "04923a7e4332423296ab247949e8d9b0" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 33713 } }
I20260926 07:31:40.450673 2019 leader_election.cc:288] T c38056c2241b498196e240cb04439711 P 04923a7e4332423296ab247949e8d9b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9e2f7f60f1324e719677220539a0858b (127.22.53.194:33065), b0f6094c72904fae960314ff0e9a7ba3 (127.22.53.193:37849)
I20260926 07:31:40.450757 2015 raft_consensus.cc:374] T c38056c2241b498196e240cb04439711 P b0f6094c72904fae960314ff0e9a7ba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:31:40.450831 2015 raft_consensus.cc:729] T c38056c2241b498196e240cb04439711 P b0f6094c72904fae960314ff0e9a7ba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0f6094c72904fae960314ff0e9a7ba3, State: Initialized, Role: FOLLOWER
I20260926 07:31:40.451321 2015 consensus_queue.cc:260] T c38056c2241b498196e240cb04439711 P b0f6094c72904fae960314ff0e9a7ba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e2f7f60f1324e719677220539a0858b" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 33065 } } peers { permanent_uuid: "b0f6094c72904fae960314ff0e9a7ba3" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 37849 } } peers { permanent_uuid: "04923a7e4332423296ab247949e8d9b0" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 33713 } }
I20260926 07:31:40.452848 2015 ts_tablet_manager.cc:1428] T c38056c2241b498196e240cb04439711 P b0f6094c72904fae960314ff0e9a7ba3: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260926 07:31:40.456620 1996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:31:40.451045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.193:37849 (local address 127.22.53.195:38237)
0926 07:31:40.451351 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0926 07:31:40.451366 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:31:40.451459 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:31:40.452765 (+ 1306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:31:40.452769 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:31:40.452793 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:31:40.452927 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:31:40.452934 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:31:40.453962 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:31:40.453965 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:31:40.455168 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:31:40.455181 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:31:40.456132 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:31:40.456136 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:31:40.456242 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:31:40.456266 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 07:31:40.456351 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 07:31:40.456442 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209}
I20260926 07:31:40.457641 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:31:40.451593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.195:38237 (local address 127.22.53.193:37849)
0926 07:31:40.451701 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0926 07:31:40.451705 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:31:40.451719 (+ 14us) server_negotiation.cc:407] Connection header received
0926 07:31:40.451764 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:31:40.451767 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:31:40.451808 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:31:40.451872 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:31:40.453066 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:31:40.453820 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:31:40.455328 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:31:40.455897 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:31:40.456010 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:31:40.456065 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 07:31:40.457426 (+ 1361us) server_negotiation.cc:299] Negotiation successful
0926 07:31:40.457521 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 07:31:40.458482 1678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c38056c2241b498196e240cb04439711" candidate_uuid: "04923a7e4332423296ab247949e8d9b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0f6094c72904fae960314ff0e9a7ba3" is_pre_election: true
I20260926 07:31:40.458832 1678 raft_consensus.cc:2454] T c38056c2241b498196e240cb04439711 P b0f6094c72904fae960314ff0e9a7ba3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 04923a7e4332423296ab247949e8d9b0 in term 0.
I20260926 07:31:40.459583 1881 leader_election.cc:302] T c38056c2241b498196e240cb04439711 P 04923a7e4332423296ab247949e8d9b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04923a7e4332423296ab247949e8d9b0, b0f6094c72904fae960314ff0e9a7ba3; no voters:
I20260926 07:31:40.460000 2019 raft_consensus.cc:2781] T c38056c2241b498196e240cb04439711 P 04923a7e4332423296ab247949e8d9b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:31:40.460083 2019 raft_consensus.cc:482] T c38056c2241b498196e240cb04439711 P 04923a7e4332423296ab247949e8d9b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:31:40.460125 2019 raft_consensus.cc:3037] T c38056c2241b498196e240cb04439711 P 04923a7e4332423296ab247949e8d9b0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:31:40.461549 2019 raft_consensus.cc:504] T c38056c2241b498196e240cb04439711 P 04923a7e4332423296ab247949e8d9b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e2f7f60f1324e719677220539a0858b" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 33065 } } peers { permanent_uuid: "b0f6094c72904fae960314ff0e9a7ba3" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 37849 } } peers { permanent_uuid: "04923a7e4332423296ab247949e8d9b0" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 33713 } }
I20260926 07:31:40.464900 2019 leader_election.cc:288] T c38056c2241b498196e240cb04439711 P 04923a7e4332423296ab247949e8d9b0 [CANDIDATE]: Term 1 election: Requested vote from peers 9e2f7f60f1324e719677220539a0858b (127.22.53.194:33065), b0f6094c72904fae960314ff0e9a7ba3 (127.22.53.193:37849)
I20260926 07:31:40.465513 1678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c38056c2241b498196e240cb04439711" candidate_uuid: "04923a7e4332423296ab247949e8d9b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0f6094c72904fae960314ff0e9a7ba3"
I20260926 07:31:40.465679 1678 raft_consensus.cc:3037] T c38056c2241b498196e240cb04439711 P b0f6094c72904fae960314ff0e9a7ba3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:31:40.467208 1678 raft_consensus.cc:2454] T c38056c2241b498196e240cb04439711 P b0f6094c72904fae960314ff0e9a7ba3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04923a7e4332423296ab247949e8d9b0 in term 1.
I20260926 07:31:40.468716 1881 leader_election.cc:302] T c38056c2241b498196e240cb04439711 P 04923a7e4332423296ab247949e8d9b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04923a7e4332423296ab247949e8d9b0, b0f6094c72904fae960314ff0e9a7ba3; no voters:
I20260926 07:31:40.468942 2019 raft_consensus.cc:2781] T c38056c2241b498196e240cb04439711 P 04923a7e4332423296ab247949e8d9b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:31:40.469208 2019 raft_consensus.cc:686] T c38056c2241b498196e240cb04439711 P 04923a7e4332423296ab247949e8d9b0 [term 1 LEADER]: Becoming Leader. State: Replica: 04923a7e4332423296ab247949e8d9b0, State: Running, Role: LEADER
I20260926 07:31:40.469609 2019 consensus_queue.cc:237] T c38056c2241b498196e240cb04439711 P 04923a7e4332423296ab247949e8d9b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e2f7f60f1324e719677220539a0858b" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 33065 } } peers { permanent_uuid: "b0f6094c72904fae960314ff0e9a7ba3" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 37849 } } peers { permanent_uuid: "04923a7e4332423296ab247949e8d9b0" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 33713 } }
I20260926 07:31:40.473377 1996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:31:40.452133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.194:33065 (local address 127.22.53.195:47611)
0926 07:31:40.456806 (+ 4673us) negotiation.cc:107] Waiting for socket to connect
0926 07:31:40.456822 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:31:40.456906 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:31:40.462236 (+ 5330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:31:40.462240 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:31:40.462270 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:31:40.462389 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:31:40.462395 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:31:40.463514 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:31:40.463518 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:31:40.464628 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:31:40.464643 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:31:40.472909 (+ 8266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:31:40.472913 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:31:40.473016 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:31:40.473041 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 07:31:40.473107 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 07:31:40.473191 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4616,"thread_start_us":124,"threads_started":1}
I20260926 07:31:40.475009 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:31:40.452587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.195:47611 (local address 127.22.53.194:33065)
0926 07:31:40.452701 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0926 07:31:40.452706 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:31:40.456993 (+ 4287us) server_negotiation.cc:407] Connection header received
0926 07:31:40.457053 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:31:40.457056 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:31:40.457109 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:31:40.457174 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:31:40.462547 (+ 5373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:31:40.463328 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:31:40.471894 (+ 8566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:31:40.472629 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:31:40.472772 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:31:40.472831 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 07:31:40.474759 (+ 1928us) server_negotiation.cc:299] Negotiation successful
0926 07:31:40.474864 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 07:31:40.475692 1808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c38056c2241b498196e240cb04439711" candidate_uuid: "04923a7e4332423296ab247949e8d9b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e2f7f60f1324e719677220539a0858b" is_pre_election: true
I20260926 07:31:40.475453 1510 catalog_manager.cc:5495] T c38056c2241b498196e240cb04439711 P 04923a7e4332423296ab247949e8d9b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 04923a7e4332423296ab247949e8d9b0 (127.22.53.195). New cstate: current_term: 1 leader_uuid: "04923a7e4332423296ab247949e8d9b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e2f7f60f1324e719677220539a0858b" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 33065 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b0f6094c72904fae960314ff0e9a7ba3" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 37849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "04923a7e4332423296ab247949e8d9b0" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 33713 } health_report { overall_health: HEALTHY } } }
I20260926 07:31:40.476042 1808 raft_consensus.cc:2454] T c38056c2241b498196e240cb04439711 P 9e2f7f60f1324e719677220539a0858b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 04923a7e4332423296ab247949e8d9b0 in term 0.
I20260926 07:31:40.476413 1811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c38056c2241b498196e240cb04439711" candidate_uuid: "04923a7e4332423296ab247949e8d9b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e2f7f60f1324e719677220539a0858b"
I20260926 07:31:40.476552 1811 raft_consensus.cc:3037] T c38056c2241b498196e240cb04439711 P 9e2f7f60f1324e719677220539a0858b [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:31:40.478164 1811 raft_consensus.cc:2454] T c38056c2241b498196e240cb04439711 P 9e2f7f60f1324e719677220539a0858b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04923a7e4332423296ab247949e8d9b0 in term 1.
I20260926 07:31:40.508903 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:31:40.502994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57164 (local address 127.22.53.195:33713)
0926 07:31:40.503137 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 07:31:40.503142 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:31:40.503403 (+ 261us) server_negotiation.cc:407] Connection header received
0926 07:31:40.503514 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:31:40.503517 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:31:40.503589 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:31:40.503697 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:31:40.504151 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:31:40.504972 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:31:40.506211 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:31:40.506532 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:31:40.506631 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:31:40.507103 (+ 472us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:31:40.507721 (+ 618us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:31:40.507859 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 07:31:40.508628 (+ 769us) server_negotiation.cc:299] Negotiation successful
0926 07:31:40.508712 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 07:31:40.517061 1811 raft_consensus.cc:1261] T c38056c2241b498196e240cb04439711 P 9e2f7f60f1324e719677220539a0858b [term 1 FOLLOWER]: Refusing update from remote peer 04923a7e4332423296ab247949e8d9b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:31:40.519167 1678 raft_consensus.cc:1261] T c38056c2241b498196e240cb04439711 P b0f6094c72904fae960314ff0e9a7ba3 [term 1 FOLLOWER]: Refusing update from remote peer 04923a7e4332423296ab247949e8d9b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:31:40.520345 2019 consensus_queue.cc:1035] T c38056c2241b498196e240cb04439711 P 04923a7e4332423296ab247949e8d9b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e2f7f60f1324e719677220539a0858b" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 33065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:31:40.522101 2019 consensus_queue.cc:1035] T c38056c2241b498196e240cb04439711 P 04923a7e4332423296ab247949e8d9b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0f6094c72904fae960314ff0e9a7ba3" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 37849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:31:40.541738 2027 mvcc.cc:205] Tried to move back new op lower bound from 7333510760507019264 to 7333510760326955008. Current Snapshot: MvccSnapshot[applied={T|T < 7333510760507019264 or (T in {7333510760507019264})}]
W20260926 07:31:40.545111 1861 tablet.cc:2369] T c38056c2241b498196e240cb04439711 P 9e2f7f60f1324e719677220539a0858b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:31:40.548197 2028 mvcc.cc:205] Tried to move back new op lower bound from 7333510760507019264 to 7333510760326955008. Current Snapshot: MvccSnapshot[applied={T|T < 7333510760507019264}]
I20260926 07:31:40.549377 1510 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45956:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:31:40.549568 1510 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45956:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:31:40.553558 2029 mvcc.cc:205] Tried to move back new op lower bound from 7333510760507019264 to 7333510760326955008. Current Snapshot: MvccSnapshot[applied={T|T < 7333510760507019264}]
I20260926 07:31:40.555163 1510 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ef8f6d76ab144909a456b7f4ff4e2bc1: Sending DeleteTablet for 3 replicas of tablet c38056c2241b498196e240cb04439711
I20260926 07:31:40.556460 1794 tablet_service.cc:1514] Processing DeleteTablet for tablet c38056c2241b498196e240cb04439711 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:31:40 UTC) from {username='kudu', principal='kudu/127.22.53.254@KRBTEST.COM'} at 127.0.0.1:60464
I20260926 07:31:40.557096 1920 tablet_service.cc:1514] Processing DeleteTablet for tablet c38056c2241b498196e240cb04439711 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:31:40 UTC) from {username='kudu', principal='kudu/127.22.53.254@KRBTEST.COM'} at 127.0.0.1:57152
I20260926 07:31:40.558105 1652 tablet_service.cc:1514] Processing DeleteTablet for tablet c38056c2241b498196e240cb04439711 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:31:40 UTC) from {username='kudu', principal='kudu/127.22.53.254@KRBTEST.COM'} at 127.0.0.1:52494
I20260926 07:31:40.559180 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 1591
I20260926 07:31:40.563926 2036 tablet_replica.cc:330] T c38056c2241b498196e240cb04439711 P 9e2f7f60f1324e719677220539a0858b: stopping tablet replica
I20260926 07:31:40.564234 2037 tablet_replica.cc:330] T c38056c2241b498196e240cb04439711 P 04923a7e4332423296ab247949e8d9b0: stopping tablet replica
I20260926 07:31:40.564886 2037 raft_consensus.cc:2229] T c38056c2241b498196e240cb04439711 P 04923a7e4332423296ab247949e8d9b0 [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:31:40.565294 2036 raft_consensus.cc:2229] T c38056c2241b498196e240cb04439711 P 9e2f7f60f1324e719677220539a0858b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:31:40.570816 2037 raft_consensus.cc:2258] T c38056c2241b498196e240cb04439711 P 04923a7e4332423296ab247949e8d9b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:31:40.571410 2036 raft_consensus.cc:2258] T c38056c2241b498196e240cb04439711 P 9e2f7f60f1324e719677220539a0858b [term 1 FOLLOWER]: Raft consensus is shut down!
W20260926 07:31:40.572103 1494 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.53.193:37849 (error 108)
I20260926 07:31:40.574399 2036 ts_tablet_manager.cc:1905] T c38056c2241b498196e240cb04439711 P 9e2f7f60f1324e719677220539a0858b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:31:40.575742 2037 ts_tablet_manager.cc:1905] T c38056c2241b498196e240cb04439711 P 04923a7e4332423296ab247949e8d9b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:31:40.579043 2037 ts_tablet_manager.cc:1918] T c38056c2241b498196e240cb04439711 P 04923a7e4332423296ab247949e8d9b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:31:40.582921 2036 ts_tablet_manager.cc:1918] T c38056c2241b498196e240cb04439711 P 9e2f7f60f1324e719677220539a0858b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260926 07:31:40.583343 1496 connection.cc:535] server connection from 127.22.53.193:50155 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 07:31:40.584038 2037 log.cc:1196] T c38056c2241b498196e240cb04439711 P 04923a7e4332423296ab247949e8d9b0: Deleting WAL directory at /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407783529717-22743-0/minicluster-data/ts-2/wal/wals/c38056c2241b498196e240cb04439711
I20260926 07:31:40.584357 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 1730
I20260926 07:31:40.586361 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:31:40.585980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.193:37849 (local address 127.0.0.1:52510)
0926 07:31:40.586147 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 07:31:40.586249 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.193:37849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260926 07:31:40.586621 1494 catalog_manager.cc:4584] TS b0f6094c72904fae960314ff0e9a7ba3 (127.22.53.193:37849): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c38056c2241b498196e240cb04439711: Network error: Client connection negotiation failed: client connection to 127.22.53.193:37849: connect: Connection refused (error 111)
I20260926 07:31:40.588033 2037 ts_tablet_manager.cc:1939] T c38056c2241b498196e240cb04439711 P 04923a7e4332423296ab247949e8d9b0: Deleting consensus metadata
I20260926 07:31:40.592051 1497 catalog_manager.cc:4841] TS 04923a7e4332423296ab247949e8d9b0 (127.22.53.195:33713): tablet c38056c2241b498196e240cb04439711 (table test-table [id=1b492a0df8eb4d35bb7fb997c66b141a]) successfully deleted
I20260926 07:31:40.593086 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 1865
I20260926 07:31:40.594362 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:31:40.593384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.194:33065 (local address 127.0.0.1:60474)
0926 07:31:40.594218 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0926 07:31:40.594273 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.194:33065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":748}
I20260926 07:31:40.601547 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 1478
I20260926 07:31:40.608839 22743 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 07:31:40.955562 22743 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 07:31:40.955741 22743 mini_ranger.cc:66] Stopping Ranger...
I20260926 07:31:41.302335 22743 mini_ranger.cc:68] Stopped Ranger
2026-09-26 07:31:41.302 UTC [1064] LOG: received smart shutdown request
2026-09-26 07:31:41.306 UTC [1064] LOG: background worker "logical replication launcher" (PID 1073) exited with exit code 1
2026-09-26 07:31:41.307 UTC [1068] LOG: shutting down
2026-09-26 07:31:41.548 UTC [1064] LOG: database system is shut down
2026-09-26T07:31:41Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (51240 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:31:41 dist-test-slave-wdjc krb5kdc[2048](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:31:41 dist-test-slave-wdjc krb5kdc[2048](info): set up 2 sockets
Sep 26 07:31:41 dist-test-slave-wdjc krb5kdc[2048](info): commencing operation
krb5kdc: starting...
W20260926 07:31:42.777851 22743 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/krb5kdc/test-user.keytab.
Sep 26 07:31:42 dist-test-slave-wdjc krb5kdc[2048](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407902, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:31:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:31:42Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.22.53.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.22.53.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.22.53.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/krb5kdc/rangeradmin_127.22.53.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.22.53.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/krb5kdc/rangeradmin_127.22.53.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.22.53.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.22.53.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.22.53.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/krb5kdc/rangerlookup_127.22.53.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.22.53.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/krb5kdc/rangerlookup_127.22.53.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.22.53.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.22.53.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.53.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/krb5kdc/HTTP_127.22.53.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.22.53.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/krb5kdc/HTTP_127.22.53.212@KRBTEST.COM.keytab.
I20260926 07:31:42.993708 22743 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskdIE4X7/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/postgres -l logfile start
2026-09-26 07:31:46.094 UTC [2078] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 07:31:46.094 UTC [2078] LOG: listening on IPv4 address "127.22.53.212", port 33151
2026-09-26 07:31:46.099 UTC [2078] LOG: listening on Unix socket "/tmp/.s.PGSQL.33151"
2026-09-26 07:31:46.137 UTC [2081] LOG: database system was shut down at 2026-09-26 07:31:44 UTC
2026-09-26 07:31:46.142 UTC [2078] LOG: database system is ready to accept connections
I20260926 07:31:47.161091 22743 mini_postgres.cc:96] Postgres bound to 33151
2026-09-26 07:31:47.167 UTC [2089] FATAL: database "slave" does not exist
127.22.53.212:33151 - accepting connections
I20260926 07:31:47.168910 22743 mini_ranger.cc:161] Starting Ranger...
I20260926 07:31:47.185751 22743 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 07:31:47.773940 22743 mini_ranger.cc:87] Created ranger Postgres database
I20260926 07:31:47.774012 22743 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ranger-admin
2026-09-26 07:31:48,091 [I] DB FLAVOR :POSTGRES
2026-09-26 07:31:48,091 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 07:31:48,091 [I] Checking connection..
2026-09-26 07:31:48,091 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdIE4X7/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdIE4X7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.53.212:33151/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 07:31:48,367 [I] Checking connection passed.
2026-09-26 07:31:48,367 [I] --------- Verifying version history table ---------
2026-09-26 07:31:48,367 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdIE4X7/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdIE4X7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.53.212:33151/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 07:31:48,628 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 07:31:48,628 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdIE4X7/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdIE4X7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.53.212:33151/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 07:31:48,912 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 07:31:48,913 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 07:31:48,913 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdIE4X7/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdIE4X7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.53.212:33151/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskdIE4X7/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 07:31:49.170 UTC [2153] WARNING: there is no transaction in progress
2026-09-26 07:31:49,176 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 07:31:49,177 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdIE4X7/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdIE4X7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.53.212:33151/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 07:31:49,441 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 07:31:49,442 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 07:31:49,442 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdIE4X7/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdIE4X7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.53.212:33151/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 07:31:49,693 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdIE4X7/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdIE4X7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.53.212:33151/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 07:31:49,950 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdIE4X7/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdIE4X7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.53.212:33151/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-wdjc.c.gcp-upstream.internal','N') ;"
2026-09-26 07:31:50,198 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdIE4X7/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdIE4X7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.53.212:33151/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 07:31:50,453 [I] Table x_portal_user does not exist in database ranger
2026-09-26 07:31:50,453 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdIE4X7/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdIE4X7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.53.212:33151/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 07:31:50,728 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 07:31:50,729 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdIE4X7/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdIE4X7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.53.212:33151/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 07:31:50,982 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdIE4X7/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdIE4X7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.53.212:33151/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 07:31:51,238 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 07:31:51,238 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdIE4X7/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdIE4X7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.53.212:33151/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskdIE4X7/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 07:31:52.315 UTC [2261] WARNING: there is no transaction in progress
2026-09-26 07:31:52.334 UTC [2261] WARNING: there is no transaction in progress
2026-09-26 07:31:52.353 UTC [2261] WARNING: there is no transaction in progress
2026-09-26 07:31:52.372 UTC [2261] WARNING: there is no transaction in progress
2026-09-26 07:31:52.393 UTC [2261] WARNING: there is no transaction in progress
2026-09-26 07:31:52.413 UTC [2261] WARNING: there is no transaction in progress
2026-09-26 07:31:52.478 UTC [2261] WARNING: there is no transaction in progress
2026-09-26 07:31:52.487 UTC [2261] WARNING: there is no transaction in progress
2026-09-26 07:31:52.503 UTC [2261] WARNING: there is no transaction in progress
2026-09-26 07:31:52.515 UTC [2261] WARNING: there is no transaction in progress
2026-09-26 07:31:52.528 UTC [2261] WARNING: there is no transaction in progress
2026-09-26 07:31:52.545 UTC [2261] WARNING: there is no transaction in progress
2026-09-26 07:31:52.557 UTC [2261] WARNING: there is no transaction in progress
2026-09-26 07:31:52.565 UTC [2261] WARNING: there is no transaction in progress
2026-09-26 07:31:52.573 UTC [2261] WARNING: there is no transaction in progress
2026-09-26 07:31:52.583 UTC [2261] WARNING: there is no transaction in progress
2026-09-26 07:31:53,393 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 07:31:53,393 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdIE4X7/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdIE4X7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.53.212:33151/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 07:31:53,674 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 07:31:53,674 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdIE4X7/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdIE4X7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.53.212:33151/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 07:31:53,944 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 07:31:53,944 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdIE4X7/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdIE4X7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.53.212:33151/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 07:31:54,198 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 07:31:54,199 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdIE4X7/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdIE4X7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.53.212:33151/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 07:31:54,452 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdIE4X7/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdIE4X7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.53.212:33151/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 07:31:54,706 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdIE4X7/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdIE4X7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.53.212:33151/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-wdjc.c.gcp-upstream.internal';"
2026-09-26 07:31:54,961 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 07:31:54,962 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdIE4X7/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdIE4X7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.53.212:33151/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 07:31:55,223 [I] DB_PATCHES have already been applied
I20260926 07:31:55.229287 22743 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ranger-admin:/tmp/dist-test-taskdIE4X7/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskdIE4X7/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskdIE4X7/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdIE4X7/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 07:31:55.229367 22743 mini_ranger.cc:193] Using host: 127.22.53.212
I20260926 07:31:55.233148 22743 mini_ranger.cc:227] Ranger admin URL: http://127.22.53.212:51963
Sep 26, 2026 7:31:55 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 7:31:55 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 7:31:55 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 7:31:55 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 7:31:55 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 7:31:56 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 7:31:56 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 07:31:56.402884 22743 mini_ranger.cc:160] Time spent starting Ranger: real 9.234s user 0.006s sys 0.005s
Sep 26, 2026 7:32:19 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 7:32:19 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 26, 2026 7:32:19 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 7:32:20 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 7:32:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 7:32:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 7:32:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 7:32:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 7:32:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 7:32:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 7:32:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 7:32:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 7:32:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 7:32:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 7:32:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 7:32:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 7:32:20 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 7:32:20 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 07:32:21.610257 22743 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.22.53.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.22.53.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.22.53.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/krb5kdc/rangerkms_127.22.53.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.22.53.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/krb5kdc/rangerkms_127.22.53.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.22.53.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.22.53.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.53.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/krb5kdc/HTTP_127.22.53.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.22.53.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/krb5kdc/HTTP_127.22.53.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 07:32:21 dist-test-slave-wdjc krb5kdc[2048](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407941, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 07:32:21.702551 22743 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 07:32:21.713639 22743 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 07:32:22.736977 22743 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 07:32:22.737071 22743 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ranger-kms
I20260926 07:32:22.737094 22743 mini_ranger_kms.cc:227] Using postgres at 127.22.53.212:33151
2026-09-26 07:32:22,933 [I] DB FLAVOR :POSTGRES
2026-09-26 07:32:22,933 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 07:32:22,934 [I] Checking connection
2026-09-26 07:32:22,934 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdIE4X7/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.53.212:33151/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 07:32:23,232 [I] connection success
2026-09-26 07:32:23,232 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 07:32:23,232 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 07:32:23,232 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdIE4X7/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.53.212:33151/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 07:32:23,498 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 07:32:23,499 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 07:32:23,499 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 07:32:23,499 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdIE4X7/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.53.212:33151/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 07:32:23,778 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 07:32:24.394855 22743 mini_ranger_kms.cc:310] Created kms service
I20260926 07:32:24.621881 22743 mini_ranger_kms.cc:326] Created kudu user
I20260926 07:32:24.670958 22743 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 07:32:24 dist-test-slave-wdjc krb5kdc[2048](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407941, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.22.53.212@KRBTEST.COM
I20260926 07:32:25.127493 22743 mini_ranger_kms.cc:384] Added ranger policy
I20260926 07:32:25.127663 22743 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ranger-kms:/tmp/dist-test-taskdIE4X7/build/debug/bin/postgresql.jar:/tmp/dist-test-taskdIE4X7/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskdIE4X7/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 07:32:25.127686 22743 mini_ranger_kms.cc:240] Using host: 127.22.53.212
I20260926 07:32:25.133350 22743 mini_ranger_kms.cc:276] Ranger KMS PID: 2442
I20260926 07:32:25.133467 22743 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.22.53.212:35201
Sep 26, 2026 7:32:26 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 7:32:26 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 7:32:26 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 7:32:26 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 7:32:27 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 7:32:27 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260926 07:32:27.509310 22743 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.807s user 0.012s sys 0.166s
I20260926 07:32:27.509447 22743 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 07:32:27.509509 22743 mini_ranger_kms.cc:401] 127.22.53.212:35201/kms/v1/keys
Sep 26 07:32:27 dist-test-slave-wdjc krb5kdc[2048](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790407947, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.22.53.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 26 07:32:28 dist-test-slave-wdjc krb5kdc[2048](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407947, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.22.53.212@KRBTEST.COM for HTTP/127.22.53.212@KRBTEST.COM
Sep 26 07:32:28 dist-test-slave-wdjc krb5kdc[2048](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407947, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.22.53.212@KRBTEST.COM for HTTP/127.22.53.212@KRBTEST.COM
Sep 26 07:32:31 dist-test-slave-wdjc krb5kdc[2048](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407951, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.22.53.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.53.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.53.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab.
Entry for principal kudu/127.22.53.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab.
I20260926 07:32:31.273442 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.53.254:33163
--webserver_interface=127.22.53.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab
--principal=kudu/127.22.53.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:43033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.53.254:33163
--ranger_config_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.53.212:35201/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:32:31.400262 2492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:32:31.400645 2492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:32:31.400694 2492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:32:31.405357 2492 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:32:31.405443 2492 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:32:31.405463 2492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:32:31.405481 2492 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:32:31.405498 2492 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:32:31.411752 2492 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:43033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/master-0/wal
--ranger_kms_url=127.22.53.212:35201/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.22.53.254:33163
--ranger_config_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab
--principal=kudu/127.22.53.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.53.254:33163
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.53.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2492
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:32:31.413057 2492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:32:31.414217 2492 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:32:31.423605 2500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:31.423605 2498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:31.423609 2497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:32:31.424041 2492 server_base.cc:1034] running on GCE node
I20260926 07:32:31.424731 2492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:32:31.426028 2492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:32:31.427258 2492 hybrid_clock.cc:648] HybridClock initialized: now 1790407951427234 us; error 39 us; skew 500 ppm
Sep 26 07:32:31 dist-test-slave-wdjc krb5kdc[2048](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407951, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:32:31.430688 2492 init.cc:377] Logged in from keytab as kudu/127.22.53.254@KRBTEST.COM (short username kudu)
I20260926 07:32:31.431826 2492 webserver.cc:460] Webserver started at http://127.22.53.254:38679/ using document root <none> and password file <none>
I20260926 07:32:31.432488 2492 fs_manager.cc:359] Metadata directory not provided
I20260926 07:32:31.432628 2492 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:32:31.433027 2492 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 07:32:31 dist-test-slave-wdjc krb5kdc[2048](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407951, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.254@KRBTEST.COM for HTTP/127.22.53.212@KRBTEST.COM
I20260926 07:32:31.490432 2492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/master-0/data/instance:
uuid: "a0c1903505c545c8ba70321f503be256"
format_stamp: "Formatted at 2026-09-26 07:32:31 on dist-test-slave-wdjc"
server_key: "0e60970f840e58dc4b08eaf2767e33de"
server_key_iv: "14c0d660b9db9dd7be3a2e192d7625a7"
server_key_version: "kuduclusterkey@0"
I20260926 07:32:31.491350 2492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/master-0/wal/instance:
uuid: "a0c1903505c545c8ba70321f503be256"
format_stamp: "Formatted at 2026-09-26 07:32:31 on dist-test-slave-wdjc"
server_key: "0e60970f840e58dc4b08eaf2767e33de"
server_key_iv: "14c0d660b9db9dd7be3a2e192d7625a7"
server_key_version: "kuduclusterkey@0"
I20260926 07:32:31.496251 2492 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 07:32:31.516700 2507 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:31.518358 2492 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:32:31.518537 2492 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/master-0/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/master-0/wal
uuid: "a0c1903505c545c8ba70321f503be256"
format_stamp: "Formatted at 2026-09-26 07:32:31 on dist-test-slave-wdjc"
server_key: "0e60970f840e58dc4b08eaf2767e33de"
server_key_iv: "14c0d660b9db9dd7be3a2e192d7625a7"
server_key_version: "kuduclusterkey@0"
I20260926 07:32:31.518695 2492 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:32:31.540894 2492 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:32:31.544335 2492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:32:31.544600 2492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:32:31.552938 2492 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.254:33163
I20260926 07:32:31.552992 2559 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.254:33163 every 8 connection(s)
I20260926 07:32:31.554123 2492 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
I20260926 07:32:31.558713 2560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:32:31.562994 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 2492
I20260926 07:32:31.563189 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/master-0/wal/instance
I20260926 07:32:31.563496 22743 external_mini_cluster.cc:1428] Setting key 244abd25ae2472f66122c0d85c5419f4
I20260926 07:32:31.567423 2560 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0c1903505c545c8ba70321f503be256: Bootstrap starting.
I20260926 07:32:31.570384 2560 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a0c1903505c545c8ba70321f503be256: Neither blocks nor log segments found. Creating new log.
I20260926 07:32:31.571322 2560 log.cc:824] T 00000000000000000000000000000000 P a0c1903505c545c8ba70321f503be256: Log is configured to *not* fsync() on all Append() calls
Sep 26 07:32:31 dist-test-slave-wdjc krb5kdc[2048](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407951, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.53.254@KRBTEST.COM
I20260926 07:32:31.573853 2560 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0c1903505c545c8ba70321f503be256: No bootstrap required, opened a new log
I20260926 07:32:31.577984 2560 raft_consensus.cc:348] T 00000000000000000000000000000000 P a0c1903505c545c8ba70321f503be256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0c1903505c545c8ba70321f503be256" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 33163 } }
I20260926 07:32:31.578293 2560 raft_consensus.cc:374] T 00000000000000000000000000000000 P a0c1903505c545c8ba70321f503be256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:32:31.578377 2560 raft_consensus.cc:729] T 00000000000000000000000000000000 P a0c1903505c545c8ba70321f503be256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0c1903505c545c8ba70321f503be256, State: Initialized, Role: FOLLOWER
I20260926 07:32:31.578908 2560 consensus_queue.cc:260] T 00000000000000000000000000000000 P a0c1903505c545c8ba70321f503be256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0c1903505c545c8ba70321f503be256" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 33163 } }
I20260926 07:32:31.579077 2560 raft_consensus.cc:388] T 00000000000000000000000000000000 P a0c1903505c545c8ba70321f503be256 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:32:31.579164 2560 raft_consensus.cc:482] T 00000000000000000000000000000000 P a0c1903505c545c8ba70321f503be256 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:32:31.579322 2560 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a0c1903505c545c8ba70321f503be256 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:32:31.580323 2563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:31.565196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37092 (local address 127.22.53.254:33163)
0926 07:32:31.565683 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:31.565702 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:31.565742 (+ 40us) server_negotiation.cc:407] Connection header received
0926 07:32:31.566445 (+ 703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:31.566464 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:31.566829 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:31.567131 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:31.568155 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:31.569295 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:31.570557 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:31.571064 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:31.571190 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:31.574024 (+ 2834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:31.574029 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:31.574044 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:32:31.574074 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:31.576999 (+ 2925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:31.577530 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:31.577536 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:31.577541 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:31.577663 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:31.577971 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:31.577978 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:31.577980 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:31.578531 (+ 551us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:31.578764 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:31.579070 (+ 306us) server_negotiation.cc:299] Negotiation successful
0926 07:32:31.579548 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":165,"threads_started":1}
I20260926 07:32:31.580399 2560 raft_consensus.cc:504] T 00000000000000000000000000000000 P a0c1903505c545c8ba70321f503be256 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0c1903505c545c8ba70321f503be256" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 33163 } }
I20260926 07:32:31.580801 2560 leader_election.cc:302] T 00000000000000000000000000000000 P a0c1903505c545c8ba70321f503be256 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a0c1903505c545c8ba70321f503be256; no voters:
I20260926 07:32:31.581274 2560 leader_election.cc:288] T 00000000000000000000000000000000 P a0c1903505c545c8ba70321f503be256 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:32:31.581351 2565 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a0c1903505c545c8ba70321f503be256 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:32:31.581612 2565 raft_consensus.cc:686] T 00000000000000000000000000000000 P a0c1903505c545c8ba70321f503be256 [term 1 LEADER]: Becoming Leader. State: Replica: a0c1903505c545c8ba70321f503be256, State: Running, Role: LEADER
I20260926 07:32:31.581944 2565 consensus_queue.cc:237] T 00000000000000000000000000000000 P a0c1903505c545c8ba70321f503be256 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0c1903505c545c8ba70321f503be256" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 33163 } }
I20260926 07:32:31.583482 2567 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0c1903505c545c8ba70321f503be256 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a0c1903505c545c8ba70321f503be256. Latest consensus state: current_term: 1 leader_uuid: "a0c1903505c545c8ba70321f503be256" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0c1903505c545c8ba70321f503be256" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 33163 } } }
I20260926 07:32:31.583617 2567 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0c1903505c545c8ba70321f503be256 [sys.catalog]: This master's current role is: LEADER
I20260926 07:32:31.583843 2560 sys_catalog.cc:564] T 00000000000000000000000000000000 P a0c1903505c545c8ba70321f503be256 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:32:31.583869 2566 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0c1903505c545c8ba70321f503be256 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a0c1903505c545c8ba70321f503be256" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0c1903505c545c8ba70321f503be256" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 33163 } } }
I20260926 07:32:31.583987 2566 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0c1903505c545c8ba70321f503be256 [sys.catalog]: This master's current role is: LEADER
I20260926 07:32:31.586360 2560 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 07:32:31.587482 2569 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:32:31.593495 2569 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:32:31.602957 2569 catalog_manager.cc:1347] Generated new cluster ID: 13da5d00d21946958a784df7938fe5a8
I20260926 07:32:31.603067 2569 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:32:31.615572 2569 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:32:31.617125 2569 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:32:31.624460 2569 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a0c1903505c545c8ba70321f503be256: Generated new TSK 0
I20260926 07:32:31.625451 2569 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.53.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.53.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.53.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.193.keytab.
Entry for principal kudu/127.22.53.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.193.keytab.
I20260926 07:32:32.377846 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.53.212:35201/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.53.193:0
--local_ip_for_outbound_sockets=127.22.53.193
--webserver_interface=127.22.53.193
--webserver_port=0
--tserver_master_addrs=127.22.53.254:33163
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.193.keytab
--principal=kudu/127.22.53.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:43033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:32:32.535772 2605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:32:32.536080 2605 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:32:32.536126 2605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:32:32.540768 2605 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:32:32.540849 2605 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:32:32.540936 2605 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.53.193
I20260926 07:32:32.546878 2605 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:43033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.22.53.212:35201/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.193.keytab
--principal=kudu/127.22.53.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.53.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.53.193
--webserver_port=0
--tserver_master_addrs=127.22.53.254:33163
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2605
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.53.193
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:32:32.548273 2605 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:32:32.549517 2605 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:32:32.561990 2612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:32.562240 2614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:32.561987 2611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:32:32.564386 2605 server_base.cc:1034] running on GCE node
I20260926 07:32:32.565008 2605 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:32:32.565943 2605 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:32:32.570322 2605 hybrid_clock.cc:648] HybridClock initialized: now 1790407952570286 us; error 52 us; skew 500 ppm
Sep 26 07:32:32 dist-test-slave-wdjc krb5kdc[2048](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407952, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:32:32.574714 2605 init.cc:377] Logged in from keytab as kudu/127.22.53.193@KRBTEST.COM (short username kudu)
I20260926 07:32:32.576071 2605 webserver.cc:460] Webserver started at http://127.22.53.193:35955/ using document root <none> and password file <none>
I20260926 07:32:32.576910 2605 fs_manager.cc:359] Metadata directory not provided
I20260926 07:32:32.577014 2605 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:32:32.577315 2605 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 07:32:32 dist-test-slave-wdjc krb5kdc[2048](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407952, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.193@KRBTEST.COM for HTTP/127.22.53.212@KRBTEST.COM
I20260926 07:32:32.594329 2605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ts-0/data/instance:
uuid: "8423875d79c04e7abe1257620bd09978"
format_stamp: "Formatted at 2026-09-26 07:32:32 on dist-test-slave-wdjc"
server_key: "919f56f39a3370a888c8bfb0b0a1f01e"
server_key_iv: "647d9b54804540259cbe14190ee9552f"
server_key_version: "kuduclusterkey@0"
I20260926 07:32:32.595157 2605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ts-0/wal/instance:
uuid: "8423875d79c04e7abe1257620bd09978"
format_stamp: "Formatted at 2026-09-26 07:32:32 on dist-test-slave-wdjc"
server_key: "919f56f39a3370a888c8bfb0b0a1f01e"
server_key_iv: "647d9b54804540259cbe14190ee9552f"
server_key_version: "kuduclusterkey@0"
I20260926 07:32:32.600683 2605 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260926 07:32:32.622219 2622 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:32.624292 2605 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260926 07:32:32.624469 2605 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ts-0/wal
uuid: "8423875d79c04e7abe1257620bd09978"
format_stamp: "Formatted at 2026-09-26 07:32:32 on dist-test-slave-wdjc"
server_key: "919f56f39a3370a888c8bfb0b0a1f01e"
server_key_iv: "647d9b54804540259cbe14190ee9552f"
server_key_version: "kuduclusterkey@0"
I20260926 07:32:32.624720 2605 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:32:32.645541 2605 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:32:32.649794 2605 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:32:32.650095 2605 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:32:32.650985 2605 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:32:32.652227 2605 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:32:32.652349 2605 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:32.652480 2605 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:32:32.652588 2605 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:32.666571 2605 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.193:37823
I20260926 07:32:32.668080 2605 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ts-0/data/info.pb
I20260926 07:32:32.675414 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 2605
I20260926 07:32:32.675570 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ts-0/wal/instance
I20260926 07:32:32.670989 2735 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.193:37823 every 8 connection(s)
I20260926 07:32:32.675885 22743 external_mini_cluster.cc:1428] Setting key bbb57cd9b0195a82a2e2959a9a8bda34
Sep 26 07:32:32 dist-test-slave-wdjc krb5kdc[2048](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407952, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.193@KRBTEST.COM for kudu/127.22.53.254@KRBTEST.COM
I20260926 07:32:32.692054 2739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:32.675530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.193:46181 (local address 127.22.53.254:33163)
0926 07:32:32.678404 (+ 2874us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:32.678410 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:32.678424 (+ 14us) server_negotiation.cc:407] Connection header received
0926 07:32:32.678466 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:32.678469 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:32.678518 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:32.678605 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:32.680446 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:32.681350 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:32.683149 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:32.683511 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:32.683601 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:32.687104 (+ 3503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:32.687108 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:32.687111 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:32:32.687143 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:32.689156 (+ 2013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:32.689953 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:32.689957 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:32.689959 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:32.690063 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:32.690541 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:32.690545 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:32.690547 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:32.690808 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:32.690898 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:32.691685 (+ 787us) server_negotiation.cc:299] Negotiation successful
0926 07:32:32.691853 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2785,"thread_start_us":109,"threads_started":1}
I20260926 07:32:32.694207 2738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:32.674159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:33163 (local address 127.22.53.193:46181)
0926 07:32:32.676755 (+ 2596us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:32.676823 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0926 07:32:32.677900 (+ 1077us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:32:32.679058 (+ 1158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:32:32.679070 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:32:32.679468 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:32:32.680239 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:32.680263 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:32.681525 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:32.681528 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:32.682976 (+ 1448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:32.682990 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:32.683857 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:32.683860 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:32.684082 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:32.684890 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:32:32.684913 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:32:32.686862 (+ 1949us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:32:32.689358 (+ 2496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:32:32.689364 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:32:32.689386 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:32:32.689813 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:32:32.690205 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:32:32.690208 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:32:32.690210 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:32:32.690417 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:32:32.690985 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:32:32.690992 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:32:32.691369 (+ 377us) client_negotiation.cc:769] Sending connection context
0926 07:32:32.692652 (+ 1283us) client_negotiation.cc:240] Negotiation successful
0926 07:32:32.693056 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2294,"thread_start_us":168,"threads_started":1}
I20260926 07:32:32.696096 2736 heartbeater.cc:347] Connected to a master server at 127.22.53.254:33163
I20260926 07:32:32.696379 2736 heartbeater.cc:464] Registering TS with master...
I20260926 07:32:32.697049 2736 heartbeater.cc:511] Master 127.22.53.254:33163 requested a full tablet report, sending...
I20260926 07:32:32.699273 2524 ts_manager.cc:194] Registered new tserver with Master: 8423875d79c04e7abe1257620bd09978 (127.22.53.193:37823)
I20260926 07:32:32.700953 2524 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.53.193@KRBTEST.COM'} at 127.22.53.193:46181
WARNING: no policy specified for kudu/127.22.53.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.53.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.53.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.194.keytab.
Entry for principal kudu/127.22.53.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.194.keytab.
I20260926 07:32:32.728398 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.53.212:35201/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.53.194:0
--local_ip_for_outbound_sockets=127.22.53.194
--webserver_interface=127.22.53.194
--webserver_port=0
--tserver_master_addrs=127.22.53.254:33163
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.194.keytab
--principal=kudu/127.22.53.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:43033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 07:32:32 dist-test-slave-wdjc krb5kdc[2048](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790407952, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260926 07:32:32.909998 2744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:32:32.910456 2744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:32:32.910576 2744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:32:32.915486 2744 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:32:32.915659 2744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:32:32.915781 2744 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.53.194
I20260926 07:32:32.923488 2744 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:43033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.22.53.212:35201/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.194.keytab
--principal=kudu/127.22.53.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.53.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.53.194
--webserver_port=0
--tserver_master_addrs=127.22.53.254:33163
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2744
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.53.194
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:32:32.925091 2744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:32:32.926200 2744 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:32:32.936691 2751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:32.936880 2753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:32.937207 2750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:32:32.937536 2744 server_base.cc:1034] running on GCE node
I20260926 07:32:32.938190 2744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:32:32.939134 2744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:32:32.940594 2744 hybrid_clock.cc:648] HybridClock initialized: now 1790407952940513 us; error 95 us; skew 500 ppm
Sep 26 07:32:32 dist-test-slave-wdjc krb5kdc[2048](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407952, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:32:32.945698 2744 init.cc:377] Logged in from keytab as kudu/127.22.53.194@KRBTEST.COM (short username kudu)
I20260926 07:32:32.947041 2744 webserver.cc:460] Webserver started at http://127.22.53.194:40359/ using document root <none> and password file <none>
I20260926 07:32:32.947880 2744 fs_manager.cc:359] Metadata directory not provided
I20260926 07:32:32.947988 2744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:32:32.948246 2744 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 07:32:32 dist-test-slave-wdjc krb5kdc[2048](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407952, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.194@KRBTEST.COM for HTTP/127.22.53.212@KRBTEST.COM
I20260926 07:32:32.976343 2744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ts-1/data/instance:
uuid: "802c7ad780794a62a16b412680f86201"
format_stamp: "Formatted at 2026-09-26 07:32:32 on dist-test-slave-wdjc"
server_key: "a686acee53506b740a39f63cf45404ef"
server_key_iv: "3368f62cfaf7859cb566f392adef6448"
server_key_version: "kuduclusterkey@0"
I20260926 07:32:32.977545 2744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ts-1/wal/instance:
uuid: "802c7ad780794a62a16b412680f86201"
format_stamp: "Formatted at 2026-09-26 07:32:32 on dist-test-slave-wdjc"
server_key: "a686acee53506b740a39f63cf45404ef"
server_key_iv: "3368f62cfaf7859cb566f392adef6448"
server_key_version: "kuduclusterkey@0"
I20260926 07:32:32.990742 2744 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.005s sys 0.000s
I20260926 07:32:33.041251 2760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:33.043573 2744 fs_manager.cc:725] Time spent opening block manager: real 0.013s user 0.003s sys 0.000s
I20260926 07:32:33.043895 2744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ts-1/wal
uuid: "802c7ad780794a62a16b412680f86201"
format_stamp: "Formatted at 2026-09-26 07:32:32 on dist-test-slave-wdjc"
server_key: "a686acee53506b740a39f63cf45404ef"
server_key_iv: "3368f62cfaf7859cb566f392adef6448"
server_key_version: "kuduclusterkey@0"
I20260926 07:32:33.044185 2744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:32:33.085659 2744 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:32:33.089851 2744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:32:33.090169 2744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:32:33.091037 2744 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:32:33.092398 2744 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:32:33.092525 2744 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:33.092674 2744 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:32:33.092741 2744 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:33.107043 2744 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.194:41073
I20260926 07:32:33.108165 2873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.194:41073 every 8 connection(s)
I20260926 07:32:33.108361 2744 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ts-1/data/info.pb
I20260926 07:32:33.112658 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 2744
I20260926 07:32:33.112825 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ts-1/wal/instance
I20260926 07:32:33.113190 22743 external_mini_cluster.cc:1428] Setting key 8cac86c4797a415e2013dc16de7e2ec5
Sep 26 07:32:33 dist-test-slave-wdjc krb5kdc[2048](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407952, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.194@KRBTEST.COM for kudu/127.22.53.254@KRBTEST.COM
I20260926 07:32:33.127560 2739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:33.110759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.194:50071 (local address 127.22.53.254:33163)
0926 07:32:33.110953 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:33.110961 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:33.112176 (+ 1215us) server_negotiation.cc:407] Connection header received
0926 07:32:33.113672 (+ 1496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:33.113676 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:33.113748 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:33.113845 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:33.115623 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:33.116387 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:33.118258 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:33.118632 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:33.118778 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:33.122391 (+ 3613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:33.122396 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:33.122400 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:32:33.122435 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:33.124705 (+ 2270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:33.125512 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:33.125516 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:33.125518 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:33.125620 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:33.126075 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:33.126079 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:33.126081 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:33.126330 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:33.126416 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:33.127211 (+ 795us) server_negotiation.cc:299] Negotiation successful
0926 07:32:33.127377 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 07:32:33.129755 2876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:33.111284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:33163 (local address 127.22.53.194:50071)
0926 07:32:33.111965 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:33.112038 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0926 07:32:33.113354 (+ 1316us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:32:33.114246 (+ 892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:32:33.114257 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:32:33.114643 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:32:33.115426 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:33.115449 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:33.116620 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:33.116624 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:33.118068 (+ 1444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:33.118082 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:33.118861 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:33.118865 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:33.119099 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:33.119924 (+ 825us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:32:33.119947 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:32:33.122119 (+ 2172us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:32:33.124903 (+ 2784us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:32:33.124911 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:32:33.124935 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:32:33.125376 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:32:33.125749 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:32:33.125752 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:32:33.125754 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:32:33.125966 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:32:33.126490 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:32:33.126498 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:32:33.126898 (+ 400us) client_negotiation.cc:769] Sending connection context
0926 07:32:33.128083 (+ 1185us) client_negotiation.cc:240] Negotiation successful
0926 07:32:33.128523 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":180,"threads_started":1}
I20260926 07:32:33.131860 2874 heartbeater.cc:347] Connected to a master server at 127.22.53.254:33163
WARNING: no policy specified for kudu/127.22.53.195@KRBTEST.COM; defaulting to no policy
I20260926 07:32:33.132154 2874 heartbeater.cc:464] Registering TS with master...
I20260926 07:32:33.132853 2874 heartbeater.cc:511] Master 127.22.53.254:33163 requested a full tablet report, sending...
I20260926 07:32:33.134423 2524 ts_manager.cc:194] Registered new tserver with Master: 802c7ad780794a62a16b412680f86201 (127.22.53.194:41073)
I20260926 07:32:33.135450 2524 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.53.194@KRBTEST.COM'} at 127.22.53.194:50071
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.53.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.53.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.195.keytab.
Entry for principal kudu/127.22.53.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.195.keytab.
I20260926 07:32:33.184180 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.53.212:35201/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.53.195:0
--local_ip_for_outbound_sockets=127.22.53.195
--webserver_interface=127.22.53.195
--webserver_port=0
--tserver_master_addrs=127.22.53.254:33163
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.195.keytab
--principal=kudu/127.22.53.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:43033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 07:32:33 dist-test-slave-wdjc krb5kdc[2048](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407952, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.254@KRBTEST.COM for HTTP/127.22.53.212@KRBTEST.COM
W20260926 07:32:33.386184 2879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:32:33.386684 2879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:32:33.386775 2879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:32:33.393224 2879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:32:33.393404 2879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:32:33.393554 2879 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.53.195
I20260926 07:32:33.401952 2879 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:43033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.22.53.212:35201/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.195.keytab
--principal=kudu/127.22.53.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.53.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.53.195
--webserver_port=0
--tserver_master_addrs=127.22.53.254:33163
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2879
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.53.195
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:32:33.403829 2879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:32:33.405568 2879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:32:33.419943 2885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:33.420607 2884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:32:33.421859 2879 server_base.cc:1034] running on GCE node
W20260926 07:32:33.423029 2887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:32:33.423779 2879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:32:33.424678 2879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:32:33.425968 2879 hybrid_clock.cc:648] HybridClock initialized: now 1790407953425901 us; error 87 us; skew 500 ppm
Sep 26 07:32:33 dist-test-slave-wdjc krb5kdc[2048](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407953, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:32:33.430070 2879 init.cc:377] Logged in from keytab as kudu/127.22.53.195@KRBTEST.COM (short username kudu)
I20260926 07:32:33.431571 2879 webserver.cc:460] Webserver started at http://127.22.53.195:43931/ using document root <none> and password file <none>
I20260926 07:32:33.432716 2879 fs_manager.cc:359] Metadata directory not provided
I20260926 07:32:33.432888 2879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:32:33.433259 2879 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 07:32:33 dist-test-slave-wdjc krb5kdc[2048](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407953, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.195@KRBTEST.COM for HTTP/127.22.53.212@KRBTEST.COM
I20260926 07:32:33.452713 2879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ts-2/data/instance:
uuid: "92c251cb8c5742f7968ea348a74dcfeb"
format_stamp: "Formatted at 2026-09-26 07:32:33 on dist-test-slave-wdjc"
server_key: "cfbd0bfec36da434b8bf6ea35c3b61ec"
server_key_iv: "d2e2fdfa5fd106e523e73dc3b52723ae"
server_key_version: "kuduclusterkey@0"
I20260926 07:32:33.453748 2879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ts-2/wal/instance:
uuid: "92c251cb8c5742f7968ea348a74dcfeb"
format_stamp: "Formatted at 2026-09-26 07:32:33 on dist-test-slave-wdjc"
server_key: "cfbd0bfec36da434b8bf6ea35c3b61ec"
server_key_iv: "d2e2fdfa5fd106e523e73dc3b52723ae"
server_key_version: "kuduclusterkey@0"
I20260926 07:32:33.459873 2879 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.002s
I20260926 07:32:33.478677 2894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:33.481110 2879 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260926 07:32:33.481307 2879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ts-2/wal
uuid: "92c251cb8c5742f7968ea348a74dcfeb"
format_stamp: "Formatted at 2026-09-26 07:32:33 on dist-test-slave-wdjc"
server_key: "cfbd0bfec36da434b8bf6ea35c3b61ec"
server_key_iv: "d2e2fdfa5fd106e523e73dc3b52723ae"
server_key_version: "kuduclusterkey@0"
I20260926 07:32:33.481467 2879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:32:33.512820 2879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:32:33.517086 2879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:32:33.517364 2879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:32:33.518227 2879 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:32:33.519517 2879 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:32:33.519603 2879 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:33.519666 2879 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:32:33.519691 2879 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:33.532786 2879 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.195:37325
I20260926 07:32:33.534603 2879 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ts-2/data/info.pb
I20260926 07:32:33.537096 3007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.195:37325 every 8 connection(s)
I20260926 07:32:33.540055 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 2879
I20260926 07:32:33.540196 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ts-2/wal/instance
I20260926 07:32:33.540560 22743 external_mini_cluster.cc:1428] Setting key e59721d4e9478e1e9295448976114bc6
Sep 26 07:32:33 dist-test-slave-wdjc krb5kdc[2048](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407953, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.195@KRBTEST.COM for kudu/127.22.53.254@KRBTEST.COM
I20260926 07:32:33.558030 2739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:33.540622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.195:34371 (local address 127.22.53.254:33163)
0926 07:32:33.540785 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:33.540789 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:33.545363 (+ 4574us) server_negotiation.cc:407] Connection header received
0926 07:32:33.546468 (+ 1105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:33.546471 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:33.546532 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:33.546590 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:33.548048 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:33.548733 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:33.549923 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:33.550204 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:33.550274 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:33.553563 (+ 3289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:33.553567 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:33.553570 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:32:33.553598 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:33.555331 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:33.556074 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:33.556078 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:33.556080 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:33.556168 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:33.556587 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:33.556594 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:33.556598 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:33.556855 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:33.556923 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:33.557686 (+ 763us) server_negotiation.cc:299] Negotiation successful
0926 07:32:33.557843 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 07:32:33.559867 3010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:33.539242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:33163 (local address 127.22.53.195:34371)
0926 07:32:33.545154 (+ 5912us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:33.545227 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0926 07:32:33.546187 (+ 960us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:32:33.546995 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:32:33.547003 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:32:33.547359 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:32:33.547892 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:33.547907 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:33.548857 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:33.548861 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:33.549761 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:33.549772 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:33.550452 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:33.550455 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:33.550604 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:33.551297 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:32:33.551321 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:32:33.553341 (+ 2020us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:32:33.555503 (+ 2162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:32:33.555510 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:32:33.555531 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:32:33.555950 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:32:33.556275 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:32:33.556277 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:32:33.556279 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:32:33.556459 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:32:33.557010 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:32:33.557016 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:32:33.557393 (+ 377us) client_negotiation.cc:769] Sending connection context
0926 07:32:33.558198 (+ 805us) client_negotiation.cc:240] Negotiation successful
0926 07:32:33.558598 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5638,"thread_start_us":186,"threads_started":1}
I20260926 07:32:33.561472 3008 heartbeater.cc:347] Connected to a master server at 127.22.53.254:33163
I20260926 07:32:33.561728 3008 heartbeater.cc:464] Registering TS with master...
I20260926 07:32:33.562311 3008 heartbeater.cc:511] Master 127.22.53.254:33163 requested a full tablet report, sending...
I20260926 07:32:33.563524 2524 ts_manager.cc:194] Registered new tserver with Master: 92c251cb8c5742f7968ea348a74dcfeb (127.22.53.195:37325)
I20260926 07:32:33.564121 2524 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.53.195@KRBTEST.COM'} at 127.22.53.195:34371
I20260926 07:32:33.570498 22743 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:32:33.571286 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 2605
I20260926 07:32:33.577884 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 2744
I20260926 07:32:33.584403 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 2879
I20260926 07:32:33.590958 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 2492
I20260926 07:32:33.597298 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.53.254:33163
--webserver_interface=127.22.53.254
--webserver_port=38679
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab
--principal=kudu/127.22.53.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:43033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.53.254:33163
--ranger_config_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.53.212:35201/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.22.53.212:35201/kms with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:32:33.722084 3011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:32:33.722414 3011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:32:33.722455 3011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:32:33.726907 3011 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:32:33.726986 3011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:32:33.727006 3011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:32:33.727023 3011 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:32:33.727039 3011 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:32:33.733057 3011 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:43033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.22.53.212:35201/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.22.53.254:33163
--ranger_config_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab
--principal=kudu/127.22.53.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.53.254:33163
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.53.254
--webserver_port=38679
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3011
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:32:33.734696 3011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:32:33.735824 3011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:32:33.745970 3019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:33.745970 3016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:33.745970 3017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:32:33.746059 3011 server_base.cc:1034] running on GCE node
I20260926 07:32:33.746958 3011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:32:33.748333 3011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:32:33.749567 3011 hybrid_clock.cc:648] HybridClock initialized: now 1790407953749547 us; error 40 us; skew 500 ppm
Sep 26 07:32:33 dist-test-slave-wdjc krb5kdc[2048](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407953, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:32:33.753808 3011 init.cc:377] Logged in from keytab as kudu/127.22.53.254@KRBTEST.COM (short username kudu)
I20260926 07:32:33.755115 3011 webserver.cc:460] Webserver started at http://127.22.53.254:38679/ using document root <none> and password file <none>
I20260926 07:32:33.755904 3011 fs_manager.cc:359] Metadata directory not provided
I20260926 07:32:33.756018 3011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 07:32:33 dist-test-slave-wdjc krb5kdc[2048](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407953, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.254@KRBTEST.COM for HTTP/127.22.53.212@KRBTEST.COM
I20260926 07:32:33.807933 3011 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 07:32:33.810761 3027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:33.812408 3011 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 07:32:33.812654 3011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/master-0/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/master-0/wal
uuid: "a0c1903505c545c8ba70321f503be256"
format_stamp: "Formatted at 2026-09-26 07:32:31 on dist-test-slave-wdjc"
server_key: "0e60970f840e58dc4b08eaf2767e33de"
server_key_iv: "14c0d660b9db9dd7be3a2e192d7625a7"
server_key_version: "kuduclusterkey@0"
I20260926 07:32:33.813328 3011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:32:33.830412 3011 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:32:33.834589 3011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:32:33.834872 3011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:32:33.843616 3079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.254:33163 every 8 connection(s)
I20260926 07:32:33.843612 3011 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.254:33163
I20260926 07:32:33.844972 3011 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
I20260926 07:32:33.846864 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 3011
I20260926 07:32:33.847368 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.53.212:35201/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.53.193:37823
--local_ip_for_outbound_sockets=127.22.53.193
--tserver_master_addrs=127.22.53.254:33163
--webserver_port=35955
--webserver_interface=127.22.53.193
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.193.keytab
--principal=kudu/127.22.53.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:43033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.22.53.212:35201/kms with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:32:33.850070 3080 sys_catalog.cc:263] Verifying existing consensus state
I20260926 07:32:33.851986 3080 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0c1903505c545c8ba70321f503be256: Bootstrap starting.
I20260926 07:32:33.871636 3080 log.cc:824] T 00000000000000000000000000000000 P a0c1903505c545c8ba70321f503be256: Log is configured to *not* fsync() on all Append() calls
I20260926 07:32:33.879796 3080 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0c1903505c545c8ba70321f503be256: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 07:32:33.880343 3080 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0c1903505c545c8ba70321f503be256: Bootstrap complete.
I20260926 07:32:33.885797 3080 raft_consensus.cc:348] T 00000000000000000000000000000000 P a0c1903505c545c8ba70321f503be256 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0c1903505c545c8ba70321f503be256" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 33163 } }
I20260926 07:32:33.886567 3080 raft_consensus.cc:729] T 00000000000000000000000000000000 P a0c1903505c545c8ba70321f503be256 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0c1903505c545c8ba70321f503be256, State: Initialized, Role: FOLLOWER
I20260926 07:32:33.887214 3080 consensus_queue.cc:260] T 00000000000000000000000000000000 P a0c1903505c545c8ba70321f503be256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0c1903505c545c8ba70321f503be256" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 33163 } }
I20260926 07:32:33.887415 3080 raft_consensus.cc:388] T 00000000000000000000000000000000 P a0c1903505c545c8ba70321f503be256 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:32:33.887503 3080 raft_consensus.cc:482] T 00000000000000000000000000000000 P a0c1903505c545c8ba70321f503be256 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:32:33.887636 3080 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a0c1903505c545c8ba70321f503be256 [term 1 FOLLOWER]: Advancing to term 2
I20260926 07:32:33.890053 3080 raft_consensus.cc:504] T 00000000000000000000000000000000 P a0c1903505c545c8ba70321f503be256 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0c1903505c545c8ba70321f503be256" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 33163 } }
I20260926 07:32:33.890460 3080 leader_election.cc:302] T 00000000000000000000000000000000 P a0c1903505c545c8ba70321f503be256 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a0c1903505c545c8ba70321f503be256; no voters:
I20260926 07:32:33.891055 3080 leader_election.cc:288] T 00000000000000000000000000000000 P a0c1903505c545c8ba70321f503be256 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 07:32:33.891155 3084 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a0c1903505c545c8ba70321f503be256 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 07:32:33.891390 3084 raft_consensus.cc:686] T 00000000000000000000000000000000 P a0c1903505c545c8ba70321f503be256 [term 2 LEADER]: Becoming Leader. State: Replica: a0c1903505c545c8ba70321f503be256, State: Running, Role: LEADER
I20260926 07:32:33.891837 3084 consensus_queue.cc:237] T 00000000000000000000000000000000 P a0c1903505c545c8ba70321f503be256 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0c1903505c545c8ba70321f503be256" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 33163 } }
I20260926 07:32:33.892596 3080 sys_catalog.cc:564] T 00000000000000000000000000000000 P a0c1903505c545c8ba70321f503be256 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:32:33.893610 3085 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0c1903505c545c8ba70321f503be256 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a0c1903505c545c8ba70321f503be256" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0c1903505c545c8ba70321f503be256" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 33163 } } }
I20260926 07:32:33.893769 3085 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0c1903505c545c8ba70321f503be256 [sys.catalog]: This master's current role is: LEADER
I20260926 07:32:33.894244 3086 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0c1903505c545c8ba70321f503be256 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a0c1903505c545c8ba70321f503be256. Latest consensus state: current_term: 2 leader_uuid: "a0c1903505c545c8ba70321f503be256" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0c1903505c545c8ba70321f503be256" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 33163 } } }
I20260926 07:32:33.894447 3086 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0c1903505c545c8ba70321f503be256 [sys.catalog]: This master's current role is: LEADER
I20260926 07:32:33.894802 3093 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:32:33.895228 3080 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 07:32:33.902040 3093 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:32:33.903254 3093 catalog_manager.cc:1259] Loaded cluster ID: 13da5d00d21946958a784df7938fe5a8
I20260926 07:32:33.903381 3093 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:32:33.906507 3093 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:32:33.907639 3093 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a0c1903505c545c8ba70321f503be256: Loaded TSK: 0
I20260926 07:32:33.909390 3093 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 07:32:34.017922 3082 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:32:34.018386 3082 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:32:34.018487 3082 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:32:34.025977 3082 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:32:34.026122 3082 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:32:34.026252 3082 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.53.193
I20260926 07:32:34.035878 3082 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:43033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.22.53.212:35201/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.193.keytab
--principal=kudu/127.22.53.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.53.193:37823
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.53.193
--webserver_port=35955
--tserver_master_addrs=127.22.53.254:33163
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3082
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.53.193
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:32:34.037550 3082 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:32:34.039029 3082 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:32:34.052847 3118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:34.053161 3120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:32:34.053617 3082 server_base.cc:1034] running on GCE node
W20260926 07:32:34.054260 3117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:32:34.054992 3082 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:32:34.056067 3082 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:32:34.063658 3082 hybrid_clock.cc:648] HybridClock initialized: now 1790407954063596 us; error 81 us; skew 500 ppm
Sep 26 07:32:34 dist-test-slave-wdjc krb5kdc[2048](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407954, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:32:34.068151 3082 init.cc:377] Logged in from keytab as kudu/127.22.53.193@KRBTEST.COM (short username kudu)
I20260926 07:32:34.069715 3082 webserver.cc:460] Webserver started at http://127.22.53.193:35955/ using document root <none> and password file <none>
I20260926 07:32:34.070624 3082 fs_manager.cc:359] Metadata directory not provided
I20260926 07:32:34.070845 3082 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 07:32:34 dist-test-slave-wdjc krb5kdc[2048](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407954, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.193@KRBTEST.COM for HTTP/127.22.53.212@KRBTEST.COM
I20260926 07:32:34.103422 3082 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 07:32:34.107174 3128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:34.109295 3082 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 07:32:34.109575 3082 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ts-0/wal
uuid: "8423875d79c04e7abe1257620bd09978"
format_stamp: "Formatted at 2026-09-26 07:32:32 on dist-test-slave-wdjc"
server_key: "919f56f39a3370a888c8bfb0b0a1f01e"
server_key_iv: "647d9b54804540259cbe14190ee9552f"
server_key_version: "kuduclusterkey@0"
I20260926 07:32:34.110549 3082 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:32:34.152472 3082 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:32:34.157341 3082 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:32:34.157675 3082 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:32:34.158605 3082 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:32:34.159953 3082 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:32:34.160072 3082 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260926 07:32:34.160243 3082 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:32:34.160342 3082 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:34.173115 3082 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.193:37823
I20260926 07:32:34.174738 3082 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ts-0/data/info.pb
I20260926 07:32:34.176406 3241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.193:37823 every 8 connection(s)
I20260926 07:32:34.178699 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 3082
I20260926 07:32:34.179476 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.53.212:35201/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.53.194:41073
--local_ip_for_outbound_sockets=127.22.53.194
--tserver_master_addrs=127.22.53.254:33163
--webserver_port=40359
--webserver_interface=127.22.53.194
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.194.keytab
--principal=kudu/127.22.53.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:43033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.22.53.212:35201/kms with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 07:32:34 dist-test-slave-wdjc krb5kdc[2048](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407954, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.193@KRBTEST.COM for kudu/127.22.53.254@KRBTEST.COM
I20260926 07:32:34.214193 3245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:34.182993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:33163 (local address 127.22.53.193:35557)
0926 07:32:34.183622 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:34.183693 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0926 07:32:34.184871 (+ 1178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:32:34.195339 (+ 10468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:32:34.195356 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:32:34.195860 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:32:34.196675 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:34.196716 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:34.199220 (+ 2504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:34.199225 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:34.200897 (+ 1672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:34.200919 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:34.204034 (+ 3115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:34.204042 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:34.204268 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:34.205069 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:32:34.205094 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:32:34.206948 (+ 1854us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:32:34.210025 (+ 3077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:32:34.210036 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:32:34.210055 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:32:34.210485 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:32:34.210998 (+ 513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:32:34.211002 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:32:34.211004 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:32:34.211196 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:32:34.212091 (+ 895us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:32:34.212098 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:32:34.212415 (+ 317us) client_negotiation.cc:769] Sending connection context
0926 07:32:34.212644 (+ 229us) client_negotiation.cc:240] Negotiation successful
0926 07:32:34.213080 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":197,"threads_started":1}
I20260926 07:32:34.214737 3246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:34.184410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.193:35557 (local address 127.22.53.254:33163)
0926 07:32:34.192987 (+ 8577us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:34.193018 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:34.193092 (+ 74us) server_negotiation.cc:407] Connection header received
0926 07:32:34.193587 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:34.193615 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:34.194206 (+ 591us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:34.194680 (+ 474us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:34.196998 (+ 2318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:34.198941 (+ 1943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:34.201100 (+ 2159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:34.203341 (+ 2241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:34.203609 (+ 268us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:34.207280 (+ 3671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:34.207288 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:34.207310 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:32:34.207351 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:34.209805 (+ 2454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:34.210657 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:34.210665 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:34.210673 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:34.210877 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:34.211311 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:34.211314 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:34.211316 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:34.211886 (+ 570us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:34.212029 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:34.213253 (+ 1224us) server_negotiation.cc:299] Negotiation successful
0926 07:32:34.213639 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8308,"thread_start_us":128,"threads_started":1}
I20260926 07:32:34.218353 3242 heartbeater.cc:347] Connected to a master server at 127.22.53.254:33163
I20260926 07:32:34.218679 3242 heartbeater.cc:464] Registering TS with master...
I20260926 07:32:34.219349 3242 heartbeater.cc:511] Master 127.22.53.254:33163 requested a full tablet report, sending...
W20260926 07:32:34.222251 3242 heartbeater.cc:644] Failed to heartbeat to 127.22.53.254:33163 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 07:32:34.222434 3242 heartbeater.cc:464] Registering TS with master...
I20260926 07:32:34.222728 3242 heartbeater.cc:511] Master 127.22.53.254:33163 requested a full tablet report, sending...
I20260926 07:32:34.223960 3242 heartbeater.cc:464] Registering TS with master...
I20260926 07:32:34.224203 3242 heartbeater.cc:511] Master 127.22.53.254:33163 requested a full tablet report, sending...
W20260926 07:32:34.225257 3242 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 07:32:34.349045 3244 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:32:34.349392 3244 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:32:34.349505 3244 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:32:34.355130 3244 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:32:34.355250 3244 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:32:34.355365 3244 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.53.194
I20260926 07:32:34.361930 3244 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:43033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.22.53.212:35201/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.194.keytab
--principal=kudu/127.22.53.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.53.194:41073
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.53.194
--webserver_port=40359
--tserver_master_addrs=127.22.53.254:33163
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3244
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.53.194
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:32:34.363521 3244 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:32:34.364995 3244 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:32:34.378698 3252 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:34.378770 3254 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:32:34.379253 3244 server_base.cc:1034] running on GCE node
W20260926 07:32:34.380784 3251 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:32:34.381407 3244 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:32:34.382300 3244 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:32:34.385622 3244 hybrid_clock.cc:648] HybridClock initialized: now 1790407954385564 us; error 73 us; skew 500 ppm
Sep 26 07:32:34 dist-test-slave-wdjc krb5kdc[2048](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407954, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:32:34.389107 3244 init.cc:377] Logged in from keytab as kudu/127.22.53.194@KRBTEST.COM (short username kudu)
I20260926 07:32:34.390286 3244 webserver.cc:460] Webserver started at http://127.22.53.194:40359/ using document root <none> and password file <none>
I20260926 07:32:34.391008 3244 fs_manager.cc:359] Metadata directory not provided
I20260926 07:32:34.391140 3244 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 07:32:34 dist-test-slave-wdjc krb5kdc[2048](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407954, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.194@KRBTEST.COM for HTTP/127.22.53.212@KRBTEST.COM
I20260926 07:32:34.411548 3244 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 07:32:34.416029 3262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:34.418244 3244 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.000s sys 0.002s
I20260926 07:32:34.418455 3244 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ts-1/wal
uuid: "802c7ad780794a62a16b412680f86201"
format_stamp: "Formatted at 2026-09-26 07:32:32 on dist-test-slave-wdjc"
server_key: "a686acee53506b740a39f63cf45404ef"
server_key_iv: "3368f62cfaf7859cb566f392adef6448"
server_key_version: "kuduclusterkey@0"
I20260926 07:32:34.419224 3244 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:32:34.438900 3244 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:32:34.444041 3244 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:32:34.444380 3244 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:32:34.445384 3244 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:32:34.446678 3244 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:32:34.446779 3244 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260926 07:32:34.446872 3244 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:32:34.446923 3244 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:34.460711 3244 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.194:41073
I20260926 07:32:34.460830 3375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.194:41073 every 8 connection(s)
I20260926 07:32:34.462412 3244 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ts-1/data/info.pb
I20260926 07:32:34.472684 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 3244
I20260926 07:32:34.473199 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.53.212:35201/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.53.195:37325
--local_ip_for_outbound_sockets=127.22.53.195
--tserver_master_addrs=127.22.53.254:33163
--webserver_port=43931
--webserver_interface=127.22.53.195
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.195.keytab
--principal=kudu/127.22.53.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:43033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.22.53.212:35201/kms with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 07:32:34 dist-test-slave-wdjc krb5kdc[2048](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407954, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.194@KRBTEST.COM for kudu/127.22.53.254@KRBTEST.COM
I20260926 07:32:34.493404 3246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:34.465273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.194:32807 (local address 127.22.53.254:33163)
0926 07:32:34.465431 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:34.465437 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:34.466815 (+ 1378us) server_negotiation.cc:407] Connection header received
0926 07:32:34.468290 (+ 1475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:34.468294 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:34.468374 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:34.468443 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:34.472511 (+ 4068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:34.473510 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:34.483166 (+ 9656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:34.483584 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:34.483699 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:34.487381 (+ 3682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:34.487385 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:34.487388 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:32:34.487421 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:34.489536 (+ 2115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:34.490418 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:34.490423 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:34.490427 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:34.490537 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:34.491085 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:34.491088 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:34.491090 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:34.491375 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:34.491476 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:34.493021 (+ 1545us) server_negotiation.cc:299] Negotiation successful
0926 07:32:34.493217 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 07:32:34.503334 3379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:34.465864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:33163 (local address 127.22.53.194:32807)
0926 07:32:34.466620 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:34.466676 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 07:32:34.467953 (+ 1277us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:32:34.470845 (+ 2892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:32:34.470859 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:32:34.471331 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:32:34.472235 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:34.472262 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:34.474600 (+ 2338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:34.474604 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:34.482882 (+ 8278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:34.482911 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:34.484018 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:34.484024 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:34.484301 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:34.485191 (+ 890us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:32:34.485222 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:32:34.487070 (+ 1848us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:32:34.489758 (+ 2688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:32:34.489768 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:32:34.489792 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:32:34.490262 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:32:34.490680 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:32:34.490683 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:32:34.490686 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:32:34.490947 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:32:34.491577 (+ 630us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:32:34.491585 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:32:34.492066 (+ 481us) client_negotiation.cc:769] Sending connection context
0926 07:32:34.495134 (+ 3068us) client_negotiation.cc:240] Negotiation successful
0926 07:32:34.501529 (+ 6395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":187,"threads_started":1}
I20260926 07:32:34.505584 3376 heartbeater.cc:347] Connected to a master server at 127.22.53.254:33163
I20260926 07:32:34.505949 3376 heartbeater.cc:464] Registering TS with master...
I20260926 07:32:34.506665 3376 heartbeater.cc:511] Master 127.22.53.254:33163 requested a full tablet report, sending...
W20260926 07:32:34.508857 3376 heartbeater.cc:644] Failed to heartbeat to 127.22.53.254:33163 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 07:32:34.509023 3376 heartbeater.cc:464] Registering TS with master...
I20260926 07:32:34.509279 3376 heartbeater.cc:511] Master 127.22.53.254:33163 requested a full tablet report, sending...
I20260926 07:32:34.510453 3376 heartbeater.cc:464] Registering TS with master...
I20260926 07:32:34.510679 3376 heartbeater.cc:511] Master 127.22.53.254:33163 requested a full tablet report, sending...
W20260926 07:32:34.511842 3376 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 07:32:34.626659 3380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:32:34.627195 3380 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:32:34.627334 3380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:32:34.634146 3380 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:32:34.634249 3380 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:32:34.634346 3380 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.53.195
I20260926 07:32:34.643193 3380 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:43033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.22.53.212:35201/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.195.keytab
--principal=kudu/127.22.53.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.53.195:37325
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.53.195
--webserver_port=43931
--tserver_master_addrs=127.22.53.254:33163
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3380
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.53.195
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:32:34.644748 3380 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:32:34.646255 3380 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:32:34.659610 3385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:34.659664 3388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:34.659610 3386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:32:34.660269 3380 server_base.cc:1034] running on GCE node
I20260926 07:32:34.660928 3380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:32:34.661954 3380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:32:34.664876 3380 hybrid_clock.cc:648] HybridClock initialized: now 1790407954664715 us; error 44 us; skew 500 ppm
Sep 26 07:32:34 dist-test-slave-wdjc krb5kdc[2048](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407954, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:32:34.668632 3380 init.cc:377] Logged in from keytab as kudu/127.22.53.195@KRBTEST.COM (short username kudu)
I20260926 07:32:34.669842 3380 webserver.cc:460] Webserver started at http://127.22.53.195:43931/ using document root <none> and password file <none>
I20260926 07:32:34.670488 3380 fs_manager.cc:359] Metadata directory not provided
I20260926 07:32:34.670605 3380 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 07:32:34 dist-test-slave-wdjc krb5kdc[2048](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407954, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.195@KRBTEST.COM for HTTP/127.22.53.212@KRBTEST.COM
I20260926 07:32:34.694979 3380 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 07:32:34.698688 3396 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:34.700914 3380 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260926 07:32:34.701112 3380 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ts-2/wal
uuid: "92c251cb8c5742f7968ea348a74dcfeb"
format_stamp: "Formatted at 2026-09-26 07:32:33 on dist-test-slave-wdjc"
server_key: "cfbd0bfec36da434b8bf6ea35c3b61ec"
server_key_iv: "d2e2fdfa5fd106e523e73dc3b52723ae"
server_key_version: "kuduclusterkey@0"
I20260926 07:32:34.701787 3380 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:32:34.723315 3380 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:32:34.727658 3380 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:32:34.727964 3380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:32:34.728859 3380 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:32:34.730072 3380 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:32:34.730209 3380 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:34.730327 3380 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:32:34.730408 3380 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:34.742321 3380 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.195:37325
I20260926 07:32:34.742484 3509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.195:37325 every 8 connection(s)
I20260926 07:32:34.743561 3380 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ts-2/data/info.pb
I20260926 07:32:34.747530 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 3380
Sep 26 07:32:34 dist-test-slave-wdjc krb5kdc[2048](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407954, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.195@KRBTEST.COM for kudu/127.22.53.254@KRBTEST.COM
I20260926 07:32:34.759994 3246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:34.745728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.195:52973 (local address 127.22.53.254:33163)
0926 07:32:34.745900 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:34.745905 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:34.746967 (+ 1062us) server_negotiation.cc:407] Connection header received
0926 07:32:34.748247 (+ 1280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:34.748251 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:34.748307 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:34.748408 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:34.749873 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:34.750413 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:34.752228 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:34.752441 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:34.752618 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:34.755367 (+ 2749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:34.755376 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:34.755383 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:32:34.755442 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:34.757275 (+ 1833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:34.757997 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:34.758001 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:34.758003 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:34.758091 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:34.758531 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:34.758534 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:34.758535 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:34.758772 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:34.758848 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:34.759653 (+ 805us) server_negotiation.cc:299] Negotiation successful
0926 07:32:34.759803 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 07:32:34.761119 3512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:34.746199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:33163 (local address 127.22.53.195:52973)
0926 07:32:34.746754 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:34.746821 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0926 07:32:34.747967 (+ 1146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:32:34.748833 (+ 866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:32:34.748843 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:32:34.749194 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:32:34.749715 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:34.749730 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:34.750549 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:34.750553 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:34.751771 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:34.751782 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:34.752673 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:34.752677 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:34.752869 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:34.753543 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:32:34.753561 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:32:34.755146 (+ 1585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:32:34.757423 (+ 2277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:32:34.757433 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:32:34.757459 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:32:34.757891 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:32:34.758185 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:32:34.758189 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:32:34.758191 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:32:34.758419 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:32:34.758936 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:32:34.758943 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:32:34.759328 (+ 385us) client_negotiation.cc:769] Sending connection context
0926 07:32:34.759576 (+ 248us) client_negotiation.cc:240] Negotiation successful
0926 07:32:34.759955 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":129,"threads_started":1}
I20260926 07:32:34.762918 3510 heartbeater.cc:347] Connected to a master server at 127.22.53.254:33163
I20260926 07:32:34.763228 3510 heartbeater.cc:464] Registering TS with master...
I20260926 07:32:34.763904 3510 heartbeater.cc:511] Master 127.22.53.254:33163 requested a full tablet report, sending...
W20260926 07:32:34.765820 3510 heartbeater.cc:644] Failed to heartbeat to 127.22.53.254:33163 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 07:32:34.765978 3510 heartbeater.cc:464] Registering TS with master...
I20260926 07:32:34.767306 3510 heartbeater.cc:511] Master 127.22.53.254:33163 requested a full tablet report, sending...
I20260926 07:32:34.768303 3510 heartbeater.cc:464] Registering TS with master...
I20260926 07:32:34.768338 3514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:34.748294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54246 (local address 127.22.53.254:33163)
0926 07:32:34.748531 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:34.748535 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:34.748675 (+ 140us) server_negotiation.cc:407] Connection header received
0926 07:32:34.748795 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:34.748798 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:34.748842 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:34.748933 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:34.750562 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:34.751308 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:34.752702 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:34.753073 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:34.753186 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:34.754958 (+ 1772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:34.754961 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:34.754966 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:32:34.754995 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:34.757688 (+ 2693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:34.758246 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:34.758250 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:34.758252 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:34.758322 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:34.759605 (+ 1283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:34.759608 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:34.759610 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:34.759831 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:34.760032 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:34.760972 (+ 940us) server_negotiation.cc:299] Negotiation successful
0926 07:32:34.768137 (+ 7165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":71,"threads_started":1}
I20260926 07:32:34.768512 3510 heartbeater.cc:511] Master 127.22.53.254:33163 requested a full tablet report, sending...
W20260926 07:32:34.769454 3510 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 26 07:32:35 dist-test-slave-wdjc krb5kdc[2048](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790407955, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:32:35.225623 3242 heartbeater.cc:464] Registering TS with master...
I20260926 07:32:35.226040 3242 heartbeater.cc:511] Master 127.22.53.254:33163 requested a full tablet report, sending...
I20260926 07:32:35.227717 3044 ts_manager.cc:194] Registered new tserver with Master: 8423875d79c04e7abe1257620bd09978 (127.22.53.193:37823)
I20260926 07:32:35.229902 3044 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.53.193@KRBTEST.COM'} at 127.22.53.193:35557
I20260926 07:32:35.512185 3376 heartbeater.cc:464] Registering TS with master...
I20260926 07:32:35.512601 3376 heartbeater.cc:511] Master 127.22.53.254:33163 requested a full tablet report, sending...
I20260926 07:32:35.513758 3044 ts_manager.cc:194] Registered new tserver with Master: 802c7ad780794a62a16b412680f86201 (127.22.53.194:41073)
I20260926 07:32:35.514631 3044 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.53.194@KRBTEST.COM'} at 127.22.53.194:32807
Sep 26 07:32:35 dist-test-slave-wdjc krb5kdc[2048](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407955, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.254@KRBTEST.COM for HTTP/127.22.53.212@KRBTEST.COM
I20260926 07:32:35.769747 3510 heartbeater.cc:464] Registering TS with master...
I20260926 07:32:35.770131 3510 heartbeater.cc:511] Master 127.22.53.254:33163 requested a full tablet report, sending...
I20260926 07:32:35.771153 3044 ts_manager.cc:194] Registered new tserver with Master: 92c251cb8c5742f7968ea348a74dcfeb (127.22.53.195:37325)
I20260926 07:32:35.772395 3044 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.53.195@KRBTEST.COM'} at 127.22.53.195:52973
I20260926 07:32:35.780808 22743 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:32:35.792681 3532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:35.782707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54258 (local address 127.22.53.254:33163)
0926 07:32:35.783023 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:35.783028 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:35.783045 (+ 17us) server_negotiation.cc:407] Connection header received
0926 07:32:35.783202 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:35.783205 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:35.783265 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:35.783385 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:35.784210 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:35.785062 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:35.786249 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:35.786611 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:35.786743 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:35.788472 (+ 1729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:35.788476 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:35.788479 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:32:35.788513 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:35.790520 (+ 2007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:35.791080 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:35.791084 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:35.791086 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:35.791196 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:35.791502 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:35.791506 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:35.791508 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:35.791787 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:35.791944 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:35.792268 (+ 324us) server_negotiation.cc:299] Negotiation successful
0926 07:32:35.792415 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":134,"threads_started":1}
I20260926 07:32:35.797868 3044 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54258:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:32:35.803155 3044 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:32:35.836192 3542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:35.823585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58182 (local address 127.22.53.193:37823)
0926 07:32:35.824697 (+ 1112us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:35.824722 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:35.826008 (+ 1286us) server_negotiation.cc:407] Connection header received
0926 07:32:35.826098 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:35.826103 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:35.826367 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:35.826508 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:32:35.827877 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:35.829161 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:35.833685 (+ 4524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:35.834553 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:35.834671 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:35.834843 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:35.835892 (+ 1049us) server_negotiation.cc:299] Negotiation successful
0926 07:32:35.836026 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":910,"thread_start_us":78,"threads_started":1}
I20260926 07:32:35.837019 3539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:35.823091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.193:37823 (local address 127.0.0.1:58182)
0926 07:32:35.825792 (+ 2701us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:35.825807 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:32:35.825907 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:32:35.827621 (+ 1714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:32:35.827623 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:32:35.827641 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:32:35.827734 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:35.827741 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:35.829317 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:35.829320 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:35.830651 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:35.830660 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:35.835466 (+ 4806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:35.835469 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:35.835624 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:35.835676 (+ 52us) client_negotiation.cc:769] Sending connection context
0926 07:32:35.836710 (+ 1034us) client_negotiation.cc:240] Negotiation successful
0926 07:32:35.836877 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2627,"thread_start_us":110,"threads_started":1}
I20260926 07:32:35.837301 3541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:35.821773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.194:41073 (local address 127.0.0.1:40310)
0926 07:32:35.824234 (+ 2461us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:35.824252 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 07:32:35.824356 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:32:35.827390 (+ 3034us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:32:35.827393 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:32:35.827411 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:32:35.827497 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:35.827503 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:35.832216 (+ 4713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:35.832220 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:35.833461 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:35.833475 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:35.836884 (+ 3409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:35.836888 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:35.836976 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:35.836997 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 07:32:35.837086 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 07:32:35.837175 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2385,"thread_start_us":97,"threads_started":1}
I20260926 07:32:35.837574 3540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:35.822204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40310 (local address 127.22.53.194:41073)
0926 07:32:35.826831 (+ 4627us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:35.826839 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:35.826860 (+ 21us) server_negotiation.cc:407] Connection header received
0926 07:32:35.826929 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:35.826934 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:35.827112 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:35.827239 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:32:35.830824 (+ 3585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:35.832054 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:35.834589 (+ 2535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:35.836565 (+ 1976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:35.836669 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:35.836821 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:35.837182 (+ 361us) server_negotiation.cc:299] Negotiation successful
0926 07:32:35.837340 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4511,"thread_start_us":121,"threads_started":1}
I20260926 07:32:35.838871 3537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:35.819801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.195:37325 (local address 127.0.0.1:45360)
0926 07:32:35.821041 (+ 1240us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:35.821099 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 07:32:35.823005 (+ 1906us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:32:35.826858 (+ 3853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:32:35.826865 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:32:35.826893 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:32:35.827051 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:35.827058 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:35.833622 (+ 6564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:35.833627 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:35.834384 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:35.834395 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:35.838461 (+ 4066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:35.838465 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:35.838565 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:35.838591 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 07:32:35.838642 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 07:32:35.838732 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":692,"thread_start_us":134,"threads_started":1}
I20260926 07:32:35.839537 3538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:35.820763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45360 (local address 127.22.53.195:37325)
0926 07:32:35.824927 (+ 4164us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:35.824934 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:35.824961 (+ 27us) server_negotiation.cc:407] Connection header received
0926 07:32:35.825050 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:35.825056 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:35.825318 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:35.825492 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:32:35.832332 (+ 6840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:35.833489 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:35.834549 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:35.835339 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:35.839064 (+ 3725us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:35.839244 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:35.839300 (+ 56us) server_negotiation.cc:299] Negotiation successful
0926 07:32:35.839403 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4050,"thread_start_us":132,"threads_started":1}
I20260926 07:32:35.841492 3310 tablet_service.cc:1467] Processing CreateTablet for tablet 72d02e859de5400daaa5b345b2bbb6e2 (DEFAULT_TABLE table=test-table [id=fcb7f94a63d749ffa6f9b9a664f6265f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:32:35.840867 3176 tablet_service.cc:1467] Processing CreateTablet for tablet 72d02e859de5400daaa5b345b2bbb6e2 (DEFAULT_TABLE table=test-table [id=fcb7f94a63d749ffa6f9b9a664f6265f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:32:35.842707 3310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72d02e859de5400daaa5b345b2bbb6e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:32:35.843034 3176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72d02e859de5400daaa5b345b2bbb6e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:32:35.843132 3444 tablet_service.cc:1467] Processing CreateTablet for tablet 72d02e859de5400daaa5b345b2bbb6e2 (DEFAULT_TABLE table=test-table [id=fcb7f94a63d749ffa6f9b9a664f6265f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:32:35.844235 3444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72d02e859de5400daaa5b345b2bbb6e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:32:35.861443 3544 tablet_bootstrap.cc:492] T 72d02e859de5400daaa5b345b2bbb6e2 P 802c7ad780794a62a16b412680f86201: Bootstrap starting.
I20260926 07:32:35.862206 3545 tablet_bootstrap.cc:492] T 72d02e859de5400daaa5b345b2bbb6e2 P 8423875d79c04e7abe1257620bd09978: Bootstrap starting.
I20260926 07:32:35.865079 3545 tablet_bootstrap.cc:654] T 72d02e859de5400daaa5b345b2bbb6e2 P 8423875d79c04e7abe1257620bd09978: Neither blocks nor log segments found. Creating new log.
I20260926 07:32:35.866597 3545 log.cc:824] T 72d02e859de5400daaa5b345b2bbb6e2 P 8423875d79c04e7abe1257620bd09978: Log is configured to *not* fsync() on all Append() calls
I20260926 07:32:35.867624 3546 tablet_bootstrap.cc:492] T 72d02e859de5400daaa5b345b2bbb6e2 P 92c251cb8c5742f7968ea348a74dcfeb: Bootstrap starting.
I20260926 07:32:35.870455 3546 tablet_bootstrap.cc:654] T 72d02e859de5400daaa5b345b2bbb6e2 P 92c251cb8c5742f7968ea348a74dcfeb: Neither blocks nor log segments found. Creating new log.
Sep 26 07:32:35 dist-test-slave-wdjc krb5kdc[2048](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407955, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.254@KRBTEST.COM for HTTP/127.22.53.212@KRBTEST.COM
I20260926 07:32:35.871692 3546 log.cc:824] T 72d02e859de5400daaa5b345b2bbb6e2 P 92c251cb8c5742f7968ea348a74dcfeb: Log is configured to *not* fsync() on all Append() calls
I20260926 07:32:35.874979 3545 tablet_bootstrap.cc:492] T 72d02e859de5400daaa5b345b2bbb6e2 P 8423875d79c04e7abe1257620bd09978: No bootstrap required, opened a new log
I20260926 07:32:35.875329 3545 ts_tablet_manager.cc:1397] T 72d02e859de5400daaa5b345b2bbb6e2 P 8423875d79c04e7abe1257620bd09978: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.000s
I20260926 07:32:35.875728 3544 tablet_bootstrap.cc:654] T 72d02e859de5400daaa5b345b2bbb6e2 P 802c7ad780794a62a16b412680f86201: Neither blocks nor log segments found. Creating new log.
I20260926 07:32:35.876682 3544 log.cc:824] T 72d02e859de5400daaa5b345b2bbb6e2 P 802c7ad780794a62a16b412680f86201: Log is configured to *not* fsync() on all Append() calls
I20260926 07:32:35.879748 3545 raft_consensus.cc:348] T 72d02e859de5400daaa5b345b2bbb6e2 P 8423875d79c04e7abe1257620bd09978 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92c251cb8c5742f7968ea348a74dcfeb" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 37325 } } peers { permanent_uuid: "802c7ad780794a62a16b412680f86201" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 41073 } } peers { permanent_uuid: "8423875d79c04e7abe1257620bd09978" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 37823 } }
I20260926 07:32:35.880257 3545 raft_consensus.cc:374] T 72d02e859de5400daaa5b345b2bbb6e2 P 8423875d79c04e7abe1257620bd09978 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:32:35.880347 3545 raft_consensus.cc:729] T 72d02e859de5400daaa5b345b2bbb6e2 P 8423875d79c04e7abe1257620bd09978 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8423875d79c04e7abe1257620bd09978, State: Initialized, Role: FOLLOWER
I20260926 07:32:35.881093 3545 consensus_queue.cc:260] T 72d02e859de5400daaa5b345b2bbb6e2 P 8423875d79c04e7abe1257620bd09978 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92c251cb8c5742f7968ea348a74dcfeb" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 37325 } } peers { permanent_uuid: "802c7ad780794a62a16b412680f86201" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 41073 } } peers { permanent_uuid: "8423875d79c04e7abe1257620bd09978" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 37823 } }
I20260926 07:32:35.883589 3546 tablet_bootstrap.cc:492] T 72d02e859de5400daaa5b345b2bbb6e2 P 92c251cb8c5742f7968ea348a74dcfeb: No bootstrap required, opened a new log
I20260926 07:32:35.883776 3544 tablet_bootstrap.cc:492] T 72d02e859de5400daaa5b345b2bbb6e2 P 802c7ad780794a62a16b412680f86201: No bootstrap required, opened a new log
I20260926 07:32:35.883921 3546 ts_tablet_manager.cc:1397] T 72d02e859de5400daaa5b345b2bbb6e2 P 92c251cb8c5742f7968ea348a74dcfeb: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.000s
I20260926 07:32:35.883991 3544 ts_tablet_manager.cc:1397] T 72d02e859de5400daaa5b345b2bbb6e2 P 802c7ad780794a62a16b412680f86201: Time spent bootstrapping tablet: real 0.023s user 0.005s sys 0.000s
I20260926 07:32:35.888365 3546 raft_consensus.cc:348] T 72d02e859de5400daaa5b345b2bbb6e2 P 92c251cb8c5742f7968ea348a74dcfeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92c251cb8c5742f7968ea348a74dcfeb" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 37325 } } peers { permanent_uuid: "802c7ad780794a62a16b412680f86201" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 41073 } } peers { permanent_uuid: "8423875d79c04e7abe1257620bd09978" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 37823 } }
I20260926 07:32:35.888371 3544 raft_consensus.cc:348] T 72d02e859de5400daaa5b345b2bbb6e2 P 802c7ad780794a62a16b412680f86201 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92c251cb8c5742f7968ea348a74dcfeb" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 37325 } } peers { permanent_uuid: "802c7ad780794a62a16b412680f86201" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 41073 } } peers { permanent_uuid: "8423875d79c04e7abe1257620bd09978" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 37823 } }
I20260926 07:32:35.888743 3544 raft_consensus.cc:374] T 72d02e859de5400daaa5b345b2bbb6e2 P 802c7ad780794a62a16b412680f86201 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:32:35.888816 3544 raft_consensus.cc:729] T 72d02e859de5400daaa5b345b2bbb6e2 P 802c7ad780794a62a16b412680f86201 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 802c7ad780794a62a16b412680f86201, State: Initialized, Role: FOLLOWER
I20260926 07:32:35.889379 3544 consensus_queue.cc:260] T 72d02e859de5400daaa5b345b2bbb6e2 P 802c7ad780794a62a16b412680f86201 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92c251cb8c5742f7968ea348a74dcfeb" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 37325 } } peers { permanent_uuid: "802c7ad780794a62a16b412680f86201" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 41073 } } peers { permanent_uuid: "8423875d79c04e7abe1257620bd09978" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 37823 } }
I20260926 07:32:35.889648 3545 ts_tablet_manager.cc:1428] T 72d02e859de5400daaa5b345b2bbb6e2 P 8423875d79c04e7abe1257620bd09978: Time spent starting tablet: real 0.014s user 0.007s sys 0.000s
I20260926 07:32:35.890066 3546 raft_consensus.cc:374] T 72d02e859de5400daaa5b345b2bbb6e2 P 92c251cb8c5742f7968ea348a74dcfeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:32:35.890105 3242 heartbeater.cc:503] Master 127.22.53.254:33163 was elected leader, sending a full tablet report...
I20260926 07:32:35.890200 3546 raft_consensus.cc:729] T 72d02e859de5400daaa5b345b2bbb6e2 P 92c251cb8c5742f7968ea348a74dcfeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92c251cb8c5742f7968ea348a74dcfeb, State: Initialized, Role: FOLLOWER
I20260926 07:32:35.890499 3544 ts_tablet_manager.cc:1428] T 72d02e859de5400daaa5b345b2bbb6e2 P 802c7ad780794a62a16b412680f86201: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260926 07:32:35.890841 3376 heartbeater.cc:503] Master 127.22.53.254:33163 was elected leader, sending a full tablet report...
I20260926 07:32:35.890678 3546 consensus_queue.cc:260] T 72d02e859de5400daaa5b345b2bbb6e2 P 92c251cb8c5742f7968ea348a74dcfeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92c251cb8c5742f7968ea348a74dcfeb" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 37325 } } peers { permanent_uuid: "802c7ad780794a62a16b412680f86201" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 41073 } } peers { permanent_uuid: "8423875d79c04e7abe1257620bd09978" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 37823 } }
I20260926 07:32:35.892853 3546 ts_tablet_manager.cc:1428] T 72d02e859de5400daaa5b345b2bbb6e2 P 92c251cb8c5742f7968ea348a74dcfeb: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260926 07:32:35.893265 3510 heartbeater.cc:503] Master 127.22.53.254:33163 was elected leader, sending a full tablet report...
W20260926 07:32:35.932725 3243 tablet.cc:2369] T 72d02e859de5400daaa5b345b2bbb6e2 P 8423875d79c04e7abe1257620bd09978: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:32:35.964440 3551 raft_consensus.cc:482] T 72d02e859de5400daaa5b345b2bbb6e2 P 802c7ad780794a62a16b412680f86201 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:32:35.964759 3551 raft_consensus.cc:504] T 72d02e859de5400daaa5b345b2bbb6e2 P 802c7ad780794a62a16b412680f86201 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92c251cb8c5742f7968ea348a74dcfeb" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 37325 } } peers { permanent_uuid: "802c7ad780794a62a16b412680f86201" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 41073 } } peers { permanent_uuid: "8423875d79c04e7abe1257620bd09978" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 37823 } }
W20260926 07:32:35.964927 3377 tablet.cc:2369] T 72d02e859de5400daaa5b345b2bbb6e2 P 802c7ad780794a62a16b412680f86201: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:32:35.965041 3552 raft_consensus.cc:482] T 72d02e859de5400daaa5b345b2bbb6e2 P 92c251cb8c5742f7968ea348a74dcfeb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:32:35.965212 3552 raft_consensus.cc:504] T 72d02e859de5400daaa5b345b2bbb6e2 P 92c251cb8c5742f7968ea348a74dcfeb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92c251cb8c5742f7968ea348a74dcfeb" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 37325 } } peers { permanent_uuid: "802c7ad780794a62a16b412680f86201" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 41073 } } peers { permanent_uuid: "8423875d79c04e7abe1257620bd09978" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 37823 } }
I20260926 07:32:35.966293 3551 leader_election.cc:288] T 72d02e859de5400daaa5b345b2bbb6e2 P 802c7ad780794a62a16b412680f86201 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92c251cb8c5742f7968ea348a74dcfeb (127.22.53.195:37325), 8423875d79c04e7abe1257620bd09978 (127.22.53.193:37823)
I20260926 07:32:35.966387 3552 leader_election.cc:288] T 72d02e859de5400daaa5b345b2bbb6e2 P 92c251cb8c5742f7968ea348a74dcfeb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 802c7ad780794a62a16b412680f86201 (127.22.53.194:41073), 8423875d79c04e7abe1257620bd09978 (127.22.53.193:37823)
I20260926 07:32:35.976277 3540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:35.969157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.195:60499 (local address 127.22.53.194:41073)
0926 07:32:35.969281 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:35.969286 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:35.969558 (+ 272us) server_negotiation.cc:407] Connection header received
0926 07:32:35.971171 (+ 1613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:35.971175 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:35.971231 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:35.971283 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:32:35.972817 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:35.973534 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:35.974895 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:35.975518 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:35.975628 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:35.975692 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:35.976022 (+ 330us) server_negotiation.cc:299] Negotiation successful
0926 07:32:35.976105 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 07:32:35.976752 3556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:35.967368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.194:41073 (local address 127.22.53.195:60499)
0926 07:32:35.969463 (+ 2095us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:35.969480 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 07:32:35.970454 (+ 974us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:32:35.971423 (+ 969us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:32:35.971426 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:32:35.971445 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:32:35.971579 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:35.971585 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:35.973713 (+ 2128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:35.973716 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:35.974750 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:35.974761 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:35.975763 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:35.975769 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:35.975875 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:35.975902 (+ 27us) client_negotiation.cc:769] Sending connection context
0926 07:32:35.976428 (+ 526us) client_negotiation.cc:240] Negotiation successful
0926 07:32:35.976567 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2021,"thread_start_us":111,"threads_started":1}
I20260926 07:32:35.977504 3330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72d02e859de5400daaa5b345b2bbb6e2" candidate_uuid: "92c251cb8c5742f7968ea348a74dcfeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "802c7ad780794a62a16b412680f86201" is_pre_election: true
I20260926 07:32:35.977774 3330 raft_consensus.cc:2454] T 72d02e859de5400daaa5b345b2bbb6e2 P 802c7ad780794a62a16b412680f86201 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92c251cb8c5742f7968ea348a74dcfeb in term 0.
I20260926 07:32:35.978307 3399 leader_election.cc:302] T 72d02e859de5400daaa5b345b2bbb6e2 P 92c251cb8c5742f7968ea348a74dcfeb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 802c7ad780794a62a16b412680f86201, 92c251cb8c5742f7968ea348a74dcfeb; no voters:
I20260926 07:32:35.978619 3552 raft_consensus.cc:2781] T 72d02e859de5400daaa5b345b2bbb6e2 P 92c251cb8c5742f7968ea348a74dcfeb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:32:35.978751 3552 raft_consensus.cc:482] T 72d02e859de5400daaa5b345b2bbb6e2 P 92c251cb8c5742f7968ea348a74dcfeb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:32:35.978811 3552 raft_consensus.cc:3037] T 72d02e859de5400daaa5b345b2bbb6e2 P 92c251cb8c5742f7968ea348a74dcfeb [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:32:35.979940 3552 raft_consensus.cc:504] T 72d02e859de5400daaa5b345b2bbb6e2 P 92c251cb8c5742f7968ea348a74dcfeb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92c251cb8c5742f7968ea348a74dcfeb" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 37325 } } peers { permanent_uuid: "802c7ad780794a62a16b412680f86201" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 41073 } } peers { permanent_uuid: "8423875d79c04e7abe1257620bd09978" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 37823 } }
I20260926 07:32:35.980419 3552 leader_election.cc:288] T 72d02e859de5400daaa5b345b2bbb6e2 P 92c251cb8c5742f7968ea348a74dcfeb [CANDIDATE]: Term 1 election: Requested vote from peers 802c7ad780794a62a16b412680f86201 (127.22.53.194:41073), 8423875d79c04e7abe1257620bd09978 (127.22.53.193:37823)
I20260926 07:32:35.980813 3330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72d02e859de5400daaa5b345b2bbb6e2" candidate_uuid: "92c251cb8c5742f7968ea348a74dcfeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "802c7ad780794a62a16b412680f86201"
I20260926 07:32:35.980945 3330 raft_consensus.cc:3037] T 72d02e859de5400daaa5b345b2bbb6e2 P 802c7ad780794a62a16b412680f86201 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:32:35.982342 3330 raft_consensus.cc:2454] T 72d02e859de5400daaa5b345b2bbb6e2 P 802c7ad780794a62a16b412680f86201 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92c251cb8c5742f7968ea348a74dcfeb in term 1.
I20260926 07:32:35.982856 3399 leader_election.cc:302] T 72d02e859de5400daaa5b345b2bbb6e2 P 92c251cb8c5742f7968ea348a74dcfeb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 802c7ad780794a62a16b412680f86201, 92c251cb8c5742f7968ea348a74dcfeb; no voters:
I20260926 07:32:35.983026 3552 raft_consensus.cc:2781] T 72d02e859de5400daaa5b345b2bbb6e2 P 92c251cb8c5742f7968ea348a74dcfeb [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:32:35.983269 3552 raft_consensus.cc:686] T 72d02e859de5400daaa5b345b2bbb6e2 P 92c251cb8c5742f7968ea348a74dcfeb [term 1 LEADER]: Becoming Leader. State: Replica: 92c251cb8c5742f7968ea348a74dcfeb, State: Running, Role: LEADER
I20260926 07:32:35.983640 3552 consensus_queue.cc:237] T 72d02e859de5400daaa5b345b2bbb6e2 P 92c251cb8c5742f7968ea348a74dcfeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92c251cb8c5742f7968ea348a74dcfeb" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 37325 } } peers { permanent_uuid: "802c7ad780794a62a16b412680f86201" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 41073 } } peers { permanent_uuid: "8423875d79c04e7abe1257620bd09978" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 37823 } }
I20260926 07:32:35.987571 3043 catalog_manager.cc:5495] T 72d02e859de5400daaa5b345b2bbb6e2 P 92c251cb8c5742f7968ea348a74dcfeb reported cstate change: term changed from 0 to 1, leader changed from <none> to 92c251cb8c5742f7968ea348a74dcfeb (127.22.53.195). New cstate: current_term: 1 leader_uuid: "92c251cb8c5742f7968ea348a74dcfeb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92c251cb8c5742f7968ea348a74dcfeb" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 37325 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "802c7ad780794a62a16b412680f86201" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 41073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8423875d79c04e7abe1257620bd09978" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 37823 } health_report { overall_health: UNKNOWN } } }
I20260926 07:32:35.995178 3542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:35.968423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.194:47159 (local address 127.22.53.193:37823)
0926 07:32:35.968841 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:35.968846 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:35.971075 (+ 2229us) server_negotiation.cc:407] Connection header received
0926 07:32:35.974964 (+ 3889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:35.974968 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:35.975026 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:35.975075 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:32:35.975793 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:35.976467 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:35.993792 (+ 17325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:35.994327 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:35.994470 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:35.994522 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:35.994916 (+ 394us) server_negotiation.cc:299] Negotiation successful
0926 07:32:35.995023 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330}
W20260926 07:32:35.995517 3511 tablet.cc:2369] T 72d02e859de5400daaa5b345b2bbb6e2 P 92c251cb8c5742f7968ea348a74dcfeb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:32:35.995705 3554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:35.966855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.193:37823 (local address 127.22.53.194:47159)
0926 07:32:35.970992 (+ 4137us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:35.971008 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:32:35.972663 (+ 1655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:32:35.975230 (+ 2567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:32:35.975233 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:32:35.975253 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:32:35.975388 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:35.975394 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:35.977396 (+ 2002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:35.977400 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:35.982386 (+ 4986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:35.982399 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:35.994629 (+ 12230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:35.994633 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:35.994750 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:35.994779 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 07:32:35.995415 (+ 636us) client_negotiation.cc:240] Negotiation successful
0926 07:32:35.995522 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4057,"thread_start_us":128,"threads_started":1}
I20260926 07:32:35.996372 3182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72d02e859de5400daaa5b345b2bbb6e2" candidate_uuid: "802c7ad780794a62a16b412680f86201" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8423875d79c04e7abe1257620bd09978" is_pre_election: true
I20260926 07:32:35.996754 3182 raft_consensus.cc:2454] T 72d02e859de5400daaa5b345b2bbb6e2 P 8423875d79c04e7abe1257620bd09978 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 802c7ad780794a62a16b412680f86201 in term 0.
I20260926 07:32:35.997356 3263 leader_election.cc:302] T 72d02e859de5400daaa5b345b2bbb6e2 P 802c7ad780794a62a16b412680f86201 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 802c7ad780794a62a16b412680f86201, 8423875d79c04e7abe1257620bd09978; no voters:
I20260926 07:32:35.997704 3551 raft_consensus.cc:2741] T 72d02e859de5400daaa5b345b2bbb6e2 P 802c7ad780794a62a16b412680f86201 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260926 07:32:35.997820 3558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:35.968490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.195:47295 (local address 127.22.53.193:37823)
0926 07:32:35.970027 (+ 1537us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:35.970035 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:35.970049 (+ 14us) server_negotiation.cc:407] Connection header received
0926 07:32:35.970105 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:35.970109 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:35.970156 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:35.970241 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:32:35.971718 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:35.972507 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:35.992685 (+ 20178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:35.993483 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:35.993620 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:35.993684 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:35.997540 (+ 3856us) server_negotiation.cc:299] Negotiation successful
0926 07:32:35.997644 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1467,"thread_start_us":139,"threads_started":1}
I20260926 07:32:35.998268 3557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:35.967771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.193:37823 (local address 127.22.53.195:47295)
0926 07:32:35.969687 (+ 1916us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:35.969700 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:32:35.969802 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:32:35.970562 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:32:35.970565 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:32:35.970578 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:32:35.970740 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:35.970747 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:35.982710 (+ 11963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:35.982713 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:35.992123 (+ 9410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:35.992143 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:35.997219 (+ 5076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:35.997223 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:35.997354 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:35.997391 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 07:32:35.998042 (+ 651us) client_negotiation.cc:240] Negotiation successful
0926 07:32:35.998146 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1857,"thread_start_us":114,"threads_started":1}
I20260926 07:32:35.998818 3182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72d02e859de5400daaa5b345b2bbb6e2" candidate_uuid: "92c251cb8c5742f7968ea348a74dcfeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8423875d79c04e7abe1257620bd09978" is_pre_election: true
I20260926 07:32:35.998939 3181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72d02e859de5400daaa5b345b2bbb6e2" candidate_uuid: "92c251cb8c5742f7968ea348a74dcfeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8423875d79c04e7abe1257620bd09978"
I20260926 07:32:35.998975 3182 raft_consensus.cc:2454] T 72d02e859de5400daaa5b345b2bbb6e2 P 8423875d79c04e7abe1257620bd09978 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92c251cb8c5742f7968ea348a74dcfeb in term 0.
I20260926 07:32:36.000905 3555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:35.967205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.195:37325 (local address 127.22.53.194:59117)
0926 07:32:35.975632 (+ 8427us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:35.975660 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 07:32:35.977263 (+ 1603us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:32:35.978134 (+ 871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:32:35.978137 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:32:35.978155 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:32:35.978265 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:35.978270 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:35.992400 (+ 14130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:35.992403 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:35.993680 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:35.993701 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:36.000388 (+ 6687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:36.000391 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:36.000508 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:36.000579 (+ 71us) client_negotiation.cc:769] Sending connection context
0926 07:32:36.000659 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 07:32:36.000756 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8344,"thread_start_us":130,"threads_started":1}
I20260926 07:32:36.000887 3538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:35.968061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.194:59117 (local address 127.22.53.195:37325)
0926 07:32:35.968188 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:35.968193 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:35.975726 (+ 7533us) server_negotiation.cc:407] Connection header received
0926 07:32:35.977796 (+ 2070us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:35.977802 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:35.977859 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:35.977919 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:32:35.980700 (+ 2781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:35.981372 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:35.999597 (+ 18225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:36.000159 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:36.000243 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:36.000282 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:36.000690 (+ 408us) server_negotiation.cc:299] Negotiation successful
0926 07:32:36.000771 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 07:32:36.001703 3446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72d02e859de5400daaa5b345b2bbb6e2" candidate_uuid: "802c7ad780794a62a16b412680f86201" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92c251cb8c5742f7968ea348a74dcfeb" is_pre_election: true
I20260926 07:32:36.024128 3538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:36.018165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45366 (local address 127.22.53.195:37325)
0926 07:32:36.018304 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:36.018308 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:36.018431 (+ 123us) server_negotiation.cc:407] Connection header received
0926 07:32:36.018648 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:36.018651 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:36.018717 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:36.018847 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:32:36.019276 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:36.020065 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:36.021430 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:36.021831 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:36.021938 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:36.022436 (+ 498us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:32:36.023124 (+ 688us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:32:36.023247 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:36.023832 (+ 585us) server_negotiation.cc:299] Negotiation successful
0926 07:32:36.023903 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 07:32:36.032480 3181 raft_consensus.cc:3037] T 72d02e859de5400daaa5b345b2bbb6e2 P 8423875d79c04e7abe1257620bd09978 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:32:36.032902 3330 raft_consensus.cc:1261] T 72d02e859de5400daaa5b345b2bbb6e2 P 802c7ad780794a62a16b412680f86201 [term 1 FOLLOWER]: Refusing update from remote peer 92c251cb8c5742f7968ea348a74dcfeb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:32:36.033737 3559 consensus_queue.cc:1035] T 72d02e859de5400daaa5b345b2bbb6e2 P 92c251cb8c5742f7968ea348a74dcfeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "802c7ad780794a62a16b412680f86201" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 41073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:32:36.034276 3181 raft_consensus.cc:1261] T 72d02e859de5400daaa5b345b2bbb6e2 P 8423875d79c04e7abe1257620bd09978 [term 1 FOLLOWER]: Refusing update from remote peer 92c251cb8c5742f7968ea348a74dcfeb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:32:36.035121 3559 consensus_queue.cc:1035] T 72d02e859de5400daaa5b345b2bbb6e2 P 92c251cb8c5742f7968ea348a74dcfeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "8423875d79c04e7abe1257620bd09978" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 37823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:32:36.049548 3562 mvcc.cc:205] Tried to move back new op lower bound from 7333510987898683392 to 7333510987712987136. Current Snapshot: MvccSnapshot[applied={T|T < 7333510987898683392 or (T in {7333510987898683392})}]
I20260926 07:32:36.052279 3563 mvcc.cc:205] Tried to move back new op lower bound from 7333510987898683392 to 7333510987712987136. Current Snapshot: MvccSnapshot[applied={T|T < 7333510987898683392}]
I20260926 07:32:36.052613 3564 mvcc.cc:205] Tried to move back new op lower bound from 7333510987898683392 to 7333510987712987136. Current Snapshot: MvccSnapshot[applied={T|T < 7333510987898683392}]
I20260926 07:32:36.057113 3043 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54258:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:32:36.057363 3043 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54258:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:32:36.060609 3043 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a0c1903505c545c8ba70321f503be256: Sending DeleteTablet for 3 replicas of tablet 72d02e859de5400daaa5b345b2bbb6e2
I20260926 07:32:36.062285 3444 tablet_service.cc:1514] Processing DeleteTablet for tablet 72d02e859de5400daaa5b345b2bbb6e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:32:36 UTC) from {username='kudu', principal='kudu/127.22.53.254@KRBTEST.COM'} at 127.0.0.1:45360
I20260926 07:32:36.062294 3176 tablet_service.cc:1514] Processing DeleteTablet for tablet 72d02e859de5400daaa5b345b2bbb6e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:32:36 UTC) from {username='kudu', principal='kudu/127.22.53.254@KRBTEST.COM'} at 127.0.0.1:58182
I20260926 07:32:36.063119 3310 tablet_service.cc:1514] Processing DeleteTablet for tablet 72d02e859de5400daaa5b345b2bbb6e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:32:36 UTC) from {username='kudu', principal='kudu/127.22.53.254@KRBTEST.COM'} at 127.0.0.1:40310
I20260926 07:32:36.064699 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 3082
I20260926 07:32:36.070163 3574 tablet_replica.cc:330] T 72d02e859de5400daaa5b345b2bbb6e2 P 802c7ad780794a62a16b412680f86201: stopping tablet replica
I20260926 07:32:36.070593 3574 raft_consensus.cc:2229] T 72d02e859de5400daaa5b345b2bbb6e2 P 802c7ad780794a62a16b412680f86201 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:32:36.071004 3574 raft_consensus.cc:2258] T 72d02e859de5400daaa5b345b2bbb6e2 P 802c7ad780794a62a16b412680f86201 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:32:36.071446 3572 tablet_replica.cc:330] T 72d02e859de5400daaa5b345b2bbb6e2 P 92c251cb8c5742f7968ea348a74dcfeb: stopping tablet replica
I20260926 07:32:36.071762 3572 raft_consensus.cc:2229] T 72d02e859de5400daaa5b345b2bbb6e2 P 92c251cb8c5742f7968ea348a74dcfeb [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:32:36.072199 3572 raft_consensus.cc:2258] T 72d02e859de5400daaa5b345b2bbb6e2 P 92c251cb8c5742f7968ea348a74dcfeb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:32:36.073997 3574 ts_tablet_manager.cc:1905] T 72d02e859de5400daaa5b345b2bbb6e2 P 802c7ad780794a62a16b412680f86201: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:32:36.077980 3572 ts_tablet_manager.cc:1905] T 72d02e859de5400daaa5b345b2bbb6e2 P 92c251cb8c5742f7968ea348a74dcfeb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:32:36.080847 3572 ts_tablet_manager.cc:1918] T 72d02e859de5400daaa5b345b2bbb6e2 P 92c251cb8c5742f7968ea348a74dcfeb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:32:36.080971 3572 log.cc:1196] T 72d02e859de5400daaa5b345b2bbb6e2 P 92c251cb8c5742f7968ea348a74dcfeb: Deleting WAL directory at /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ts-2/wal/wals/72d02e859de5400daaa5b345b2bbb6e2
I20260926 07:32:36.081327 3572 ts_tablet_manager.cc:1939] T 72d02e859de5400daaa5b345b2bbb6e2 P 92c251cb8c5742f7968ea348a74dcfeb: Deleting consensus metadata
I20260926 07:32:36.082784 3030 catalog_manager.cc:4841] TS 92c251cb8c5742f7968ea348a74dcfeb (127.22.53.195:37325): tablet 72d02e859de5400daaa5b345b2bbb6e2 (table test-table [id=fcb7f94a63d749ffa6f9b9a664f6265f]) successfully deleted
I20260926 07:32:36.088886 3574 ts_tablet_manager.cc:1918] T 72d02e859de5400daaa5b345b2bbb6e2 P 802c7ad780794a62a16b412680f86201: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:32:36.089046 3574 log.cc:1196] T 72d02e859de5400daaa5b345b2bbb6e2 P 802c7ad780794a62a16b412680f86201: Deleting WAL directory at /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407783529717-22743-0/minicluster-data/ts-1/wal/wals/72d02e859de5400daaa5b345b2bbb6e2
I20260926 07:32:36.089429 3574 ts_tablet_manager.cc:1939] T 72d02e859de5400daaa5b345b2bbb6e2 P 802c7ad780794a62a16b412680f86201: Deleting consensus metadata
I20260926 07:32:36.091034 3030 catalog_manager.cc:4841] TS 802c7ad780794a62a16b412680f86201 (127.22.53.194:41073): tablet 72d02e859de5400daaa5b345b2bbb6e2 (table test-table [id=fcb7f94a63d749ffa6f9b9a664f6265f]) successfully deleted
W20260926 07:32:36.091984 3028 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.53.193:37823 (error 108)
I20260926 07:32:36.093031 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 3244
I20260926 07:32:36.100168 3537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:36.094638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.193:37823 (local address 127.0.0.1:58188)
0926 07:32:36.099921 (+ 5283us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:36.100047 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.193:37823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5144}
W20260926 07:32:36.100492 3028 catalog_manager.cc:4584] TS 8423875d79c04e7abe1257620bd09978 (127.22.53.193:37823): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 72d02e859de5400daaa5b345b2bbb6e2: Network error: Client connection negotiation failed: client connection to 127.22.53.193:37823: connect: Connection refused (error 111)
I20260926 07:32:36.103049 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 3380
I20260926 07:32:36.110281 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 3011
I20260926 07:32:36.119765 22743 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 07:32:36.465927 22743 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 07:32:36.466084 22743 mini_ranger.cc:66] Stopping Ranger...
I20260926 07:32:36.812427 22743 mini_ranger.cc:68] Stopped Ranger
2026-09-26 07:32:36.812 UTC [2078] LOG: received smart shutdown request
2026-09-26 07:32:36.816 UTC [2078] LOG: background worker "logical replication launcher" (PID 2087) exited with exit code 1
2026-09-26 07:32:36.817 UTC [2082] LOG: shutting down
2026-09-26 07:32:37.008 UTC [2078] LOG: database system is shut down
2026-09-26T07:32:37Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (55458 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407783529717-22743-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:32:37 dist-test-slave-wdjc krb5kdc[3584](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:32:37 dist-test-slave-wdjc krb5kdc[3584](info): set up 2 sockets
Sep 26 07:32:37 dist-test-slave-wdjc krb5kdc[3584](info): commencing operation
krb5kdc: starting...
W20260926 07:32:38.213816 22743 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.006s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407783529717-22743-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407783529717-22743-0/krb5kdc/test-user.keytab.
Sep 26 07:32:38 dist-test-slave-wdjc krb5kdc[3584](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:32:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:32:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.53.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.53.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.53.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab.
Entry for principal kudu/127.22.53.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab.
I20260926 07:32:38.356948 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407783529717-22743-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407783529717-22743-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407783529717-22743-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.53.254:35695
--webserver_interface=127.22.53.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab
--principal=kudu/127.22.53.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:45755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.53.254:35695,127.22.53.253:40619,127.22.53.252:39885
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:32:38.486024 3597 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:32:38.486393 3597 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:32:38.486440 3597 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:32:38.491015 3597 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:32:38.491115 3597 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:32:38.491137 3597 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:32:38.491154 3597 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:32:38.491174 3597 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:32:38.499459 3597 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:45755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407783529717-22743-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407783529717-22743-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.53.254:35695,127.22.53.253:40619,127.22.53.252:39885
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab
--principal=kudu/127.22.53.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.53.254:35695
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.53.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3597
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407783529717-22743-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:32:38.500782 3597 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:32:38.501959 3597 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:32:38.513147 3605 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:38.513149 3602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:38.513147 3603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:32:38.513451 3597 server_base.cc:1034] running on GCE node
I20260926 07:32:38.514325 3597 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:32:38.515717 3597 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:32:38.516920 3597 hybrid_clock.cc:648] HybridClock initialized: now 1790407958516886 us; error 63 us; skew 500 ppm
Sep 26 07:32:38 dist-test-slave-wdjc krb5kdc[3584](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407958, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:32:38.520457 3597 init.cc:377] Logged in from keytab as kudu/127.22.53.254@KRBTEST.COM (short username kudu)
I20260926 07:32:38.521649 3597 webserver.cc:460] Webserver started at http://127.22.53.254:42219/ using document root <none> and password file <none>
I20260926 07:32:38.522305 3597 fs_manager.cc:359] Metadata directory not provided
I20260926 07:32:38.522380 3597 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:32:38.522575 3597 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:32:38.525250 3597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407783529717-22743-0/minicluster-data/master-0/data/instance:
uuid: "d38b788c833747109549919f0de72509"
format_stamp: "Formatted at 2026-09-26 07:32:38 on dist-test-slave-wdjc"
server_key: "8d2a32cf7edf1b1c8a20747b3f906c25"
server_key_iv: "77d7abdb834d74a036c21e406a1ce832"
server_key_version: "encryptionkey@0"
I20260926 07:32:38.526005 3597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407783529717-22743-0/minicluster-data/master-0/wal/instance:
uuid: "d38b788c833747109549919f0de72509"
format_stamp: "Formatted at 2026-09-26 07:32:38 on dist-test-slave-wdjc"
server_key: "8d2a32cf7edf1b1c8a20747b3f906c25"
server_key_iv: "77d7abdb834d74a036c21e406a1ce832"
server_key_version: "encryptionkey@0"
I20260926 07:32:38.530573 3597 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260926 07:32:38.533651 3612 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:38.535477 3597 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 07:32:38.535635 3597 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407783529717-22743-0/minicluster-data/master-0/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407783529717-22743-0/minicluster-data/master-0/wal
uuid: "d38b788c833747109549919f0de72509"
format_stamp: "Formatted at 2026-09-26 07:32:38 on dist-test-slave-wdjc"
server_key: "8d2a32cf7edf1b1c8a20747b3f906c25"
server_key_iv: "77d7abdb834d74a036c21e406a1ce832"
server_key_version: "encryptionkey@0"
I20260926 07:32:38.536062 3597 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407783529717-22743-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407783529717-22743-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407783529717-22743-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:32:38.556296 3597 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:32:38.560232 3597 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:32:38.560425 3597 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:32:38.569027 3597 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.254:35695
I20260926 07:32:38.569085 3664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.254:35695 every 8 connection(s)
I20260926 07:32:38.570274 3597 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
I20260926 07:32:38.573906 3665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:32:38.575984 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 3597
I20260926 07:32:38.576267 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407783529717-22743-0/minicluster-data/master-0/wal/instance
I20260926 07:32:38.577018 22743 external_mini_cluster.cc:1428] Setting key a70018e554f53136a00a5e5115ba460f
I20260926 07:32:38.579015 3665 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.53.254" port: 35695 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.22.53.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.53.253@KRBTEST.COM" created.
Sep 26 07:32:38 dist-test-slave-wdjc krb5kdc[3584](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407958, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.254@KRBTEST.COM for kudu/127.22.53.254@KRBTEST.COM
I20260926 07:32:38.600201 3669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:38.582414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:35695 (local address 127.0.0.1:60138)
0926 07:32:38.583271 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:38.583315 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 07:32:38.584332 (+ 1017us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:32:38.585229 (+ 897us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:32:38.585235 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:32:38.585722 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:32:38.586017 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:38.586033 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:38.587383 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:38.587389 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:38.588210 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:38.588222 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:38.589295 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:38.589299 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:38.589500 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:38.590355 (+ 855us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:32:38.590382 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:32:38.592962 (+ 2580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:32:38.596450 (+ 3488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:32:38.596461 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:32:38.596472 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:32:38.596913 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:32:38.597391 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:32:38.597393 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:32:38.597394 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:32:38.597520 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:32:38.598498 (+ 978us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:32:38.598504 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:32:38.598679 (+ 175us) client_negotiation.cc:769] Sending connection context
0926 07:32:38.598874 (+ 195us) client_negotiation.cc:240] Negotiation successful
0926 07:32:38.599208 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":680,"mutex_wait_us":30,"spinlock_wait_cycles":55680,"thread_start_us":213,"threads_started":1}
I20260926 07:32:38.600311 3668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:38.582414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60138 (local address 127.22.53.254:35695)
0926 07:32:38.583004 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:38.583020 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:38.583455 (+ 435us) server_negotiation.cc:407] Connection header received
0926 07:32:38.584768 (+ 1313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:38.584773 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:38.584992 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:38.585221 (+ 229us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:38.586198 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:38.587234 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:38.588362 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:38.589030 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:38.589359 (+ 329us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:38.593311 (+ 3952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:38.593324 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:38.593348 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:32:38.593400 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:38.596226 (+ 2826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:38.597065 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:38.597075 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:38.597079 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:38.597264 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:38.597628 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:38.597633 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:38.597635 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:38.598329 (+ 694us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:38.598505 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:38.599009 (+ 504us) server_negotiation.cc:299] Negotiation successful
0926 07:32:38.599330 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":123,"server-negotiator.queue_time_us":424,"thread_start_us":213,"threads_started":1}
I20260926 07:32:38.603235 3665 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.53.253" port: 40619 } has no permanent_uuid. Determining permanent_uuid...
I20260926 07:32:38.604167 3669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:38.603853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.253:40619 (local address 127.0.0.1:36000)
0926 07:32:38.603996 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:38.604091 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.253:40619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260926 07:32:38.604480 3613 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.53.253:40619: connect: Connection refused (error 111)
I20260926 07:32:38.605657 3669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:38.605395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.253:40619 (local address 127.0.0.1:36006)
0926 07:32:38.605527 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:38.605584 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.253:40619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260926 07:32:38.605865 3665 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.53.253:40619: Network error: Client connection negotiation failed: client connection to 127.22.53.253:40619: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.53.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.253.keytab.
Entry for principal kudu/127.22.53.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.253.keytab.
I20260926 07:32:38.612841 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407783529717-22743-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407783529717-22743-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407783529717-22743-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407783529717-22743-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.53.253:40619
--webserver_interface=127.22.53.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.253.keytab
--principal=kudu/127.22.53.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:45755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.53.254:35695,127.22.53.253:40619,127.22.53.252:39885
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:32:38.655232 3665 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.53.253" port: 40619 } attempt: 1
I20260926 07:32:38.656291 3669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:38.655902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.253:40619 (local address 127.0.0.1:36022)
0926 07:32:38.656128 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:38.656201 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.253:40619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260926 07:32:38.657243 3669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:38.656955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.253:40619 (local address 127.0.0.1:36026)
0926 07:32:38.657098 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:38.657179 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.253:40619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260926 07:32:38.657506 3665 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.53.253:40619: Network error: Client connection negotiation failed: client connection to 127.22.53.253:40619: connect: Connection refused (error 111)
I20260926 07:32:38.725714 3665 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.53.253" port: 40619 } attempt: 2
I20260926 07:32:38.726660 3669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:38.726348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.253:40619 (local address 127.0.0.1:36034)
0926 07:32:38.726505 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:38.726578 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.253:40619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260926 07:32:38.727464 3669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:38.727225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.253:40619 (local address 127.0.0.1:36048)
0926 07:32:38.727342 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:38.727402 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.253:40619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260926 07:32:38.727635 3665 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.53.253:40619: Network error: Client connection negotiation failed: client connection to 127.22.53.253:40619: connect: Connection refused (error 111)
W20260926 07:32:38.738126 3672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:32:38.738468 3672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:32:38.738513 3672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:32:38.743119 3672 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:32:38.743206 3672 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:32:38.743227 3672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:32:38.743247 3672 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:32:38.743263 3672 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:32:38.749423 3672 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:45755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407783529717-22743-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407783529717-22743-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.22.53.254:35695,127.22.53.253:40619,127.22.53.252:39885
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.253.keytab
--principal=kudu/127.22.53.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.53.253:40619
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407783529717-22743-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.22.53.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3672
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407783529717-22743-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:32:38.750747 3672 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:32:38.751885 3672 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:32:38.761648 3677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:32:38.761835 3672 server_base.cc:1034] running on GCE node
W20260926 07:32:38.761669 3680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:38.761641 3678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:32:38.762717 3672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:32:38.764137 3672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:32:38.765398 3672 hybrid_clock.cc:648] HybridClock initialized: now 1790407958765363 us; error 59 us; skew 500 ppm
Sep 26 07:32:38 dist-test-slave-wdjc krb5kdc[3584](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407958, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:32:38.769243 3672 init.cc:377] Logged in from keytab as kudu/127.22.53.253@KRBTEST.COM (short username kudu)
I20260926 07:32:38.770488 3672 webserver.cc:460] Webserver started at http://127.22.53.253:36575/ using document root <none> and password file <none>
I20260926 07:32:38.771353 3672 fs_manager.cc:359] Metadata directory not provided
I20260926 07:32:38.771461 3672 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:32:38.771726 3672 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:32:38.774688 3672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407783529717-22743-0/minicluster-data/master-1/data/instance:
uuid: "240849e070f34383a33ca7b0129d0874"
format_stamp: "Formatted at 2026-09-26 07:32:38 on dist-test-slave-wdjc"
server_key: "15990c4be2abe6ea4a56e93178ad7177"
server_key_iv: "b8d2ec9b79769b609f1db522909c593d"
server_key_version: "encryptionkey@0"
I20260926 07:32:38.775492 3672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407783529717-22743-0/minicluster-data/master-1/wal/instance:
uuid: "240849e070f34383a33ca7b0129d0874"
format_stamp: "Formatted at 2026-09-26 07:32:38 on dist-test-slave-wdjc"
server_key: "15990c4be2abe6ea4a56e93178ad7177"
server_key_iv: "b8d2ec9b79769b609f1db522909c593d"
server_key_version: "encryptionkey@0"
I20260926 07:32:38.780942 3672 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 07:32:38.784446 3687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:38.786490 3672 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 07:32:38.786747 3672 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407783529717-22743-0/minicluster-data/master-1/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407783529717-22743-0/minicluster-data/master-1/wal
uuid: "240849e070f34383a33ca7b0129d0874"
format_stamp: "Formatted at 2026-09-26 07:32:38 on dist-test-slave-wdjc"
server_key: "15990c4be2abe6ea4a56e93178ad7177"
server_key_iv: "b8d2ec9b79769b609f1db522909c593d"
server_key_version: "encryptionkey@0"
I20260926 07:32:38.786919 3672 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407783529717-22743-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407783529717-22743-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407783529717-22743-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:32:38.805433 3672 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:32:38.809585 3672 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:32:38.809859 3672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:32:38.818806 3665 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.53.253" port: 40619 } attempt: 3
I20260926 07:32:38.819581 3672 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.253:40619
I20260926 07:32:38.819610 3739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.253:40619 every 8 connection(s)
I20260926 07:32:38.819802 3669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:38.819465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.253:40619 (local address 127.0.0.1:36054)
0926 07:32:38.819606 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:38.819681 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.253:40619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260926 07:32:38.821096 3672 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407783529717-22743-0/minicluster-data/master-1/data/info.pb
I20260926 07:32:38.822060 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 3672
I20260926 07:32:38.822242 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407783529717-22743-0/minicluster-data/master-1/wal/instance
I20260926 07:32:38.822582 22743 external_mini_cluster.cc:1428] Setting key 3fb32661c881ccc0607cc31b52875b5d
I20260926 07:32:38.827430 3740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 07:32:38 dist-test-slave-wdjc krb5kdc[3584](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407958, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.254@KRBTEST.COM for kudu/127.22.53.253@KRBTEST.COM
WARNING: no policy specified for kudu/127.22.53.252@KRBTEST.COM; defaulting to no policy
I20260926 07:32:38.833328 3740 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.53.254" port: 35695 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.53.252@KRBTEST.COM" created.
I20260926 07:32:38.837148 3669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:38.820446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.253:40619 (local address 127.0.0.1:36056)
0926 07:32:38.820600 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:38.820616 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:32:38.820760 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:32:38.823689 (+ 2929us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:32:38.823697 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:32:38.823970 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:32:38.824200 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:38.824215 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:38.826507 (+ 2292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:38.826513 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:38.827393 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:38.827412 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:38.828772 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:38.828777 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:38.828918 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:38.829662 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:32:38.829684 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:32:38.831574 (+ 1890us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:32:38.834652 (+ 3078us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:32:38.834656 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:32:38.834659 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:32:38.835008 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:32:38.835506 (+ 498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:32:38.835509 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:32:38.835510 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:32:38.835598 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:32:38.836451 (+ 853us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:32:38.836455 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:32:38.836643 (+ 188us) client_negotiation.cc:769] Sending connection context
0926 07:32:38.836776 (+ 133us) client_negotiation.cc:240] Negotiation successful
0926 07:32:38.836963 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260926 07:32:38.839080 3741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:38.821424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36056 (local address 127.22.53.253:40619)
0926 07:32:38.821911 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:38.821931 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:38.821984 (+ 53us) server_negotiation.cc:407] Connection header received
0926 07:32:38.822865 (+ 881us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:38.822887 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:38.823304 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:38.823661 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:38.824439 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:38.826293 (+ 1854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:38.827577 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:38.828571 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:38.828794 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:38.831876 (+ 3082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:38.831886 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:38.831910 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:32:38.831957 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:38.834485 (+ 2528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:38.835166 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:38.835176 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:38.835186 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:38.835385 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:38.835709 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:38.835713 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:38.835715 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:38.836346 (+ 631us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:38.836469 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:38.836947 (+ 478us) server_negotiation.cc:299] Negotiation successful
0926 07:32:38.837935 (+ 988us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":164,"threads_started":1}
I20260926 07:32:38.841821 3665 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.53.252" port: 39885 } has no permanent_uuid. Determining permanent_uuid...
I20260926 07:32:38.842758 3669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:38.842435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.252:39885 (local address 127.0.0.1:57188)
0926 07:32:38.842610 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:38.842668 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.252:39885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260926 07:32:38.843609 3669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:38.843328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.252:39885 (local address 127.0.0.1:57200)
0926 07:32:38.843472 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:38.843533 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.252:39885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260926 07:32:38.843891 3665 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.53.252:39885: Network error: Client connection negotiation failed: client connection to 127.22.53.252:39885: connect: Connection refused (error 111)
Sep 26 07:32:38 dist-test-slave-wdjc krb5kdc[3584](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407958, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.253@KRBTEST.COM for kudu/127.22.53.254@KRBTEST.COM
I20260926 07:32:38.849643 3744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:38.835606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:35695 (local address 127.0.0.1:60144)
0926 07:32:38.836109 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:38.836165 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 07:32:38.836286 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:32:38.837820 (+ 1534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:32:38.837826 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:32:38.838301 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:32:38.838470 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:38.838478 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:38.839633 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:38.839640 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:38.840912 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:38.840925 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:38.841787 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:38.841791 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:38.841921 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:38.842627 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:32:38.842640 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:32:38.844350 (+ 1710us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:32:38.846777 (+ 2427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:32:38.846797 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:32:38.846807 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:32:38.847246 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:32:38.847796 (+ 550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:32:38.847799 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:32:38.847801 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:32:38.847953 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:32:38.848976 (+ 1023us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:32:38.848983 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:32:38.849120 (+ 137us) client_negotiation.cc:769] Sending connection context
0926 07:32:38.849260 (+ 140us) client_negotiation.cc:240] Negotiation successful
0926 07:32:38.849465 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":167,"threads_started":1}
I20260926 07:32:38.850221 3668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:38.837383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60144 (local address 127.22.53.254:35695)
0926 07:32:38.837568 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:38.837572 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:38.837585 (+ 13us) server_negotiation.cc:407] Connection header received
0926 07:32:38.837629 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:38.837632 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:38.837670 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:38.837776 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:38.838633 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:38.839429 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:38.841070 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:38.841466 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:38.841606 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:38.844609 (+ 3003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:38.844614 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:38.844621 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:32:38.844655 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:38.846509 (+ 1854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:38.847563 (+ 1054us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:38.847567 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:38.847568 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:38.847671 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:38.848156 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:38.848161 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:38.848163 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:38.848820 (+ 657us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:38.848952 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:38.849970 (+ 1018us) server_negotiation.cc:299] Negotiation successful
0926 07:32:38.850105 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260926 07:32:38.851446 3740 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.53.253" port: 40619 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.53.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.252.keytab.
Entry for principal kudu/127.22.53.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.252.keytab.
I20260926 07:32:38.856070 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407783529717-22743-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407783529717-22743-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407783529717-22743-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407783529717-22743-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.53.252:39885
--webserver_interface=127.22.53.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.252.keytab
--principal=kudu/127.22.53.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:45755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.53.254:35695,127.22.53.253:40619,127.22.53.252:39885
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 07:32:38 dist-test-slave-wdjc krb5kdc[3584](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407958, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.253@KRBTEST.COM for kudu/127.22.53.253@KRBTEST.COM
I20260926 07:32:38.860734 3741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:38.852075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36058 (local address 127.22.53.253:40619)
0926 07:32:38.852191 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:38.852194 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:38.852230 (+ 36us) server_negotiation.cc:407] Connection header received
0926 07:32:38.852351 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:38.852353 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:38.852391 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:38.852453 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:38.853014 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:38.853657 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:38.854612 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:38.854911 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:38.854994 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:38.857369 (+ 2375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:38.857373 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:38.857375 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:32:38.857395 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:38.858886 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:38.859391 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:38.859396 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:38.859398 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:38.859478 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:38.859757 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:38.859762 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:38.859764 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:38.859967 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:38.860078 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:38.860362 (+ 284us) server_negotiation.cc:299] Negotiation successful
0926 07:32:38.860601 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":93,"server-negotiator.queue_time_us":44}
I20260926 07:32:38.860747 3744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:38.852001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.253:40619 (local address 127.0.0.1:36058)
0926 07:32:38.852153 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:38.852169 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:32:38.852248 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:32:38.852520 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:32:38.852522 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:32:38.852789 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:32:38.852898 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:38.852904 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:38.853795 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:38.853799 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:38.854489 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:38.854501 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:38.855051 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:38.855055 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:38.855183 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:38.855810 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:32:38.855822 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:32:38.857202 (+ 1380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:32:38.859049 (+ 1847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:32:38.859052 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:32:38.859055 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:32:38.859284 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:32:38.859581 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:32:38.859584 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:32:38.859586 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:32:38.859665 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:32:38.860075 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:32:38.860078 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:32:38.860186 (+ 108us) client_negotiation.cc:769] Sending connection context
0926 07:32:38.860306 (+ 120us) client_negotiation.cc:240] Negotiation successful
0926 07:32:38.860483 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260926 07:32:38.861557 3740 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.53.252" port: 39885 } has no permanent_uuid. Determining permanent_uuid...
I20260926 07:32:38.862442 3744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:38.862137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.252:39885 (local address 127.0.0.1:57202)
0926 07:32:38.862269 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:38.862370 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.252:39885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260926 07:32:38.863065 3688 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.53.252:39885: connect: Connection refused (error 111)
I20260926 07:32:38.864112 3744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:38.863872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.252:39885 (local address 127.0.0.1:57218)
0926 07:32:38.863991 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:38.864048 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.252:39885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260926 07:32:38.864264 3740 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.53.252:39885: Network error: Client connection negotiation failed: client connection to 127.22.53.252:39885: connect: Connection refused (error 111)
I20260926 07:32:38.875207 3665 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.53.252" port: 39885 } attempt: 1
I20260926 07:32:38.876111 3669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:38.875797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.252:39885 (local address 127.0.0.1:57232)
0926 07:32:38.875952 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:38.876025 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.252:39885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260926 07:32:38.877405 3669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:38.877040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.252:39885 (local address 127.0.0.1:57234)
0926 07:32:38.877210 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:38.877290 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.252:39885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260926 07:32:38.877640 3665 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.53.252:39885: Network error: Client connection negotiation failed: client connection to 127.22.53.252:39885: connect: Connection refused (error 111)
I20260926 07:32:38.913585 3740 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.53.252" port: 39885 } attempt: 1
I20260926 07:32:38.914529 3744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:38.914221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.252:39885 (local address 127.0.0.1:57242)
0926 07:32:38.914386 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:38.914448 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.252:39885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260926 07:32:38.915292 3744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:38.915076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.252:39885 (local address 127.0.0.1:57258)
0926 07:32:38.915186 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:38.915239 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.252:39885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260926 07:32:38.915421 3740 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.53.252:39885: Network error: Client connection negotiation failed: client connection to 127.22.53.252:39885: connect: Connection refused (error 111)
I20260926 07:32:38.952852 3665 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.53.252" port: 39885 } attempt: 2
I20260926 07:32:38.953778 3669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:38.953440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.252:39885 (local address 127.0.0.1:57264)
0926 07:32:38.953617 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:38.953682 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.252:39885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260926 07:32:38.954468 3669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:38.954253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.252:39885 (local address 127.0.0.1:57274)
0926 07:32:38.954361 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:38.954412 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.252:39885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260926 07:32:38.954594 3665 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.53.252:39885: Network error: Client connection negotiation failed: client connection to 127.22.53.252:39885: connect: Connection refused (error 111)
W20260926 07:32:38.977754 3746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:32:38.978138 3746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:32:38.978229 3746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:32:38.982625 3746 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:32:38.982717 3746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:32:38.982755 3746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:32:38.982816 3746 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:32:38.982854 3746 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:32:38.983628 3740 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.53.252" port: 39885 } attempt: 2
I20260926 07:32:38.984572 3744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:38.984227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.252:39885 (local address 127.0.0.1:57278)
0926 07:32:38.984387 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:38.984455 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.252:39885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260926 07:32:38.985265 3744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:38.985048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.252:39885 (local address 127.0.0.1:57284)
0926 07:32:38.985153 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:38.985200 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.252:39885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260926 07:32:38.985388 3740 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.53.252:39885: Network error: Client connection negotiation failed: client connection to 127.22.53.252:39885: connect: Connection refused (error 111)
I20260926 07:32:38.988993 3746 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:45755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407783529717-22743-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407783529717-22743-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.22.53.254:35695,127.22.53.253:40619,127.22.53.252:39885
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.252.keytab
--principal=kudu/127.22.53.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.53.252:39885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407783529717-22743-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.22.53.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3746
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407783529717-22743-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:32:38.990201 3746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:32:38.991338 3746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:32:39.001094 3753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:39.001080 3755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:39.001082 3752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:32:39.001593 3746 server_base.cc:1034] running on GCE node
I20260926 07:32:39.002259 3746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:32:39.003509 3746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:32:39.004654 3746 hybrid_clock.cc:648] HybridClock initialized: now 1790407959004642 us; error 31 us; skew 500 ppm
Sep 26 07:32:39 dist-test-slave-wdjc krb5kdc[3584](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407959, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:32:39.008450 3746 init.cc:377] Logged in from keytab as kudu/127.22.53.252@KRBTEST.COM (short username kudu)
I20260926 07:32:39.010100 3746 webserver.cc:460] Webserver started at http://127.22.53.252:33513/ using document root <none> and password file <none>
I20260926 07:32:39.010982 3746 fs_manager.cc:359] Metadata directory not provided
I20260926 07:32:39.011089 3746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:32:39.011387 3746 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:32:39.014504 3746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407783529717-22743-0/minicluster-data/master-2/data/instance:
uuid: "56e96067b6d6486fa7ea34958450eb8b"
format_stamp: "Formatted at 2026-09-26 07:32:39 on dist-test-slave-wdjc"
server_key: "9c47a8d46894a874a4e078ca1cfe1a75"
server_key_iv: "a817acb3a9071bcc4e9a7b8ea8f63520"
server_key_version: "encryptionkey@0"
I20260926 07:32:39.015278 3746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407783529717-22743-0/minicluster-data/master-2/wal/instance:
uuid: "56e96067b6d6486fa7ea34958450eb8b"
format_stamp: "Formatted at 2026-09-26 07:32:39 on dist-test-slave-wdjc"
server_key: "9c47a8d46894a874a4e078ca1cfe1a75"
server_key_iv: "a817acb3a9071bcc4e9a7b8ea8f63520"
server_key_version: "encryptionkey@0"
I20260926 07:32:39.019876 3746 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 07:32:39.022907 3762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:39.024412 3746 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 07:32:39.024643 3746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407783529717-22743-0/minicluster-data/master-2/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407783529717-22743-0/minicluster-data/master-2/wal
uuid: "56e96067b6d6486fa7ea34958450eb8b"
format_stamp: "Formatted at 2026-09-26 07:32:39 on dist-test-slave-wdjc"
server_key: "9c47a8d46894a874a4e078ca1cfe1a75"
server_key_iv: "a817acb3a9071bcc4e9a7b8ea8f63520"
server_key_version: "encryptionkey@0"
I20260926 07:32:39.024803 3746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407783529717-22743-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407783529717-22743-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407783529717-22743-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:32:39.036499 3746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:32:39.039873 3746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:32:39.040087 3746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:32:39.047758 3746 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.252:39885
I20260926 07:32:39.047780 3814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.252:39885 every 8 connection(s)
I20260926 07:32:39.048841 3746 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407783529717-22743-0/minicluster-data/master-2/data/info.pb
I20260926 07:32:39.051673 3815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:32:39.053783 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 3746
I20260926 07:32:39.053773 3665 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.53.252" port: 39885 } attempt: 3
I20260926 07:32:39.053921 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407783529717-22743-0/minicluster-data/master-2/wal/instance
I20260926 07:32:39.054204 22743 external_mini_cluster.cc:1428] Setting key b66d82fe42be825e8eca52e036d4305f
I20260926 07:32:39.057072 3815 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.53.254" port: 35695 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 07:32:39 dist-test-slave-wdjc krb5kdc[3584](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407958, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.254@KRBTEST.COM for kudu/127.22.53.252@KRBTEST.COM
Sep 26 07:32:39 dist-test-slave-wdjc krb5kdc[3584](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.53.252@KRBTEST.COM
Sep 26 07:32:39 dist-test-slave-wdjc krb5kdc[3584](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407959, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.252@KRBTEST.COM for kudu/127.22.53.254@KRBTEST.COM
I20260926 07:32:39.070209 3669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:39.054414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.252:39885 (local address 127.0.0.1:57298)
0926 07:32:39.054639 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:39.054661 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 07:32:39.054780 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:32:39.057542 (+ 2762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:32:39.057546 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:32:39.057771 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:32:39.057934 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:39.057943 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:39.060599 (+ 2656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:39.060603 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:39.061436 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:39.061448 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:39.062593 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:39.062597 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:39.062679 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:39.063481 (+ 802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:32:39.063500 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:32:39.065164 (+ 1664us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:32:39.067747 (+ 2583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:32:39.067750 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:32:39.067753 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:32:39.068029 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:32:39.068510 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:32:39.068516 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:32:39.068520 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:32:39.068649 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:32:39.069514 (+ 865us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:32:39.069520 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:32:39.069636 (+ 116us) client_negotiation.cc:769] Sending connection context
0926 07:32:39.069819 (+ 183us) client_negotiation.cc:240] Negotiation successful
0926 07:32:39.069982 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132}
I20260926 07:32:39.070768 3668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:39.058991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60150 (local address 127.22.53.254:35695)
0926 07:32:39.059114 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:39.059119 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:39.059842 (+ 723us) server_negotiation.cc:407] Connection header received
0926 07:32:39.060031 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:39.060034 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:39.060079 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:39.060127 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:39.061141 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:39.061879 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:39.063066 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:39.063459 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:39.063590 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:39.066779 (+ 3189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:39.066783 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:39.066787 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:32:39.066822 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:39.068848 (+ 2026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:39.069391 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:39.069396 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:39.069398 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:39.069485 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:39.069762 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:39.069768 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:39.069772 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:39.070001 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:39.070097 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:39.070513 (+ 416us) server_negotiation.cc:299] Negotiation successful
0926 07:32:39.070627 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 07:32:39.071802 3820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:39.058968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:35695 (local address 127.0.0.1:60150)
0926 07:32:39.059713 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:39.059767 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 07:32:39.059930 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:32:39.060294 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:32:39.060303 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:32:39.060849 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:32:39.060997 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:39.061005 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:39.062013 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:39.062016 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:39.062946 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:39.062955 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:39.063632 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:39.063636 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:39.063761 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:39.064258 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:32:39.064275 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:32:39.066634 (+ 2359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:32:39.068992 (+ 2358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:32:39.069001 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:32:39.069008 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:32:39.069284 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:32:39.069590 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:32:39.069597 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:32:39.069599 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:32:39.069672 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:32:39.070109 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:32:39.070115 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:32:39.070256 (+ 141us) client_negotiation.cc:769] Sending connection context
0926 07:32:39.070897 (+ 641us) client_negotiation.cc:240] Negotiation successful
0926 07:32:39.071094 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"mutex_wait_us":3,"thread_start_us":149,"threads_started":1}
I20260926 07:32:39.071802 3818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:39.055271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57298 (local address 127.22.53.252:39885)
0926 07:32:39.055752 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:39.055768 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:39.055805 (+ 37us) server_negotiation.cc:407] Connection header received
0926 07:32:39.056644 (+ 839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:39.056669 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:39.057156 (+ 487us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:39.058179 (+ 1023us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:39.058744 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:39.060406 (+ 1662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:39.061601 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:39.062455 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:39.062633 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:39.065320 (+ 2687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:39.065326 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:39.065338 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:32:39.065359 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:39.067601 (+ 2242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:39.068171 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:39.068180 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:39.068189 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:39.068365 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:39.068775 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:39.068778 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:39.068780 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:39.069398 (+ 618us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:39.069548 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:39.069965 (+ 417us) server_negotiation.cc:299] Negotiation successful
0926 07:32:39.070516 (+ 551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":553,"server-negotiator.queue_time_us":339,"spinlock_wait_cycles":7808,"thread_start_us":146,"threads_started":1}
I20260926 07:32:39.072070 3819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:39.055733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57314 (local address 127.22.53.252:39885)
0926 07:32:39.055990 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:39.055998 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:39.056009 (+ 11us) server_negotiation.cc:407] Connection header received
0926 07:32:39.056650 (+ 641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:39.056669 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:39.057156 (+ 487us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:39.058401 (+ 1245us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:39.058744 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:39.060399 (+ 1655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:39.061646 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:39.062460 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:39.062949 (+ 489us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:39.066079 (+ 3130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:39.066083 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:39.066086 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:32:39.066129 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:39.069019 (+ 2890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:39.070426 (+ 1407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:39.070430 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:39.070432 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:39.070518 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:39.070894 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:39.070899 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:39.070901 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:39.071118 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:39.071326 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:39.071698 (+ 372us) server_negotiation.cc:299] Negotiation successful
0926 07:32:39.071849 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":92,"server-negotiator.queue_time_us":199,"spinlock_wait_cycles":15232,"thread_start_us":80,"threads_started":1}
I20260926 07:32:39.074170 3815 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.53.253" port: 40619 } has no permanent_uuid. Determining permanent_uuid...
I20260926 07:32:39.076588 3740 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.53.252" port: 39885 } attempt: 3
I20260926 07:32:39.076915 3665 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d38b788c833747109549919f0de72509: Bootstrap starting.
I20260926 07:32:39.080117 3665 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d38b788c833747109549919f0de72509: Neither blocks nor log segments found. Creating new log.
Sep 26 07:32:39 dist-test-slave-wdjc krb5kdc[3584](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407959, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.252@KRBTEST.COM for kudu/127.22.53.253@KRBTEST.COM
I20260926 07:32:39.081166 3665 log.cc:824] T 00000000000000000000000000000000 P d38b788c833747109549919f0de72509: Log is configured to *not* fsync() on all Append() calls
Sep 26 07:32:39 dist-test-slave-wdjc krb5kdc[3584](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.53.254@KRBTEST.COM
Sep 26 07:32:39 dist-test-slave-wdjc krb5kdc[3584](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407958, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.253@KRBTEST.COM for kudu/127.22.53.252@KRBTEST.COM
I20260926 07:32:39.083925 3665 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d38b788c833747109549919f0de72509: No bootstrap required, opened a new log
I20260926 07:32:39.086186 3741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:39.074813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36072 (local address 127.22.53.253:40619)
0926 07:32:39.074928 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:39.074932 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:39.075012 (+ 80us) server_negotiation.cc:407] Connection header received
0926 07:32:39.075153 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:39.075156 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:39.075218 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:39.075304 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:39.075954 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:39.076756 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:39.077874 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:39.078282 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:39.078363 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:39.082072 (+ 3709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:39.082076 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:39.082079 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:32:39.082106 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:39.084094 (+ 1988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:39.084634 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:39.084638 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:39.084641 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:39.084734 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:39.085080 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:39.085083 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:39.085085 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:39.085374 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:39.085491 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:39.085840 (+ 349us) server_negotiation.cc:299] Negotiation successful
0926 07:32:39.085988 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33,"spinlock_wait_cycles":3840}
I20260926 07:32:39.086786 3820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:39.074719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.253:40619 (local address 127.0.0.1:36072)
0926 07:32:39.074934 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:39.074950 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:32:39.075047 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:32:39.075319 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:32:39.075322 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:32:39.075513 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:32:39.075622 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:39.075628 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:39.076897 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:39.076900 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:39.077733 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:39.077745 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:39.078523 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:39.078526 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:39.078632 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:39.079218 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:32:39.079239 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:32:39.081897 (+ 2658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:32:39.084196 (+ 2299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:32:39.084199 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:32:39.084201 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:32:39.084449 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:32:39.084863 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:32:39.084866 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:32:39.084870 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:32:39.084969 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:32:39.085491 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:32:39.085494 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:32:39.085612 (+ 118us) client_negotiation.cc:769] Sending connection context
0926 07:32:39.086444 (+ 832us) client_negotiation.cc:240] Negotiation successful
0926 07:32:39.086615 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149}
I20260926 07:32:39.086910 3744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:39.077012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.252:39885 (local address 127.0.0.1:57330)
0926 07:32:39.077292 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:39.077307 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:32:39.077381 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:32:39.077875 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:32:39.077878 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:32:39.078086 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:32:39.078190 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:39.078197 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:39.079284 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:39.079287 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:39.080081 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:39.080092 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:39.080760 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:39.080763 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:39.080846 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:39.081333 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:32:39.081350 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:32:39.082976 (+ 1626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:32:39.085331 (+ 2355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:32:39.085334 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:32:39.085336 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:32:39.085584 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:32:39.085886 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:32:39.085891 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:32:39.085895 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:32:39.085979 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:32:39.086426 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:32:39.086429 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:32:39.086534 (+ 105us) client_negotiation.cc:769] Sending connection context
0926 07:32:39.086616 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 07:32:39.086749 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"spinlock_wait_cycles":768}
I20260926 07:32:39.087441 3818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:39.077541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57330 (local address 127.22.53.252:39885)
0926 07:32:39.077657 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:39.077662 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:39.077674 (+ 12us) server_negotiation.cc:407] Connection header received
0926 07:32:39.077708 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:39.077710 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:39.077750 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:39.078310 (+ 560us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:39.078414 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:39.079148 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:39.080236 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:39.080611 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:39.081078 (+ 467us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:39.083155 (+ 2077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:39.083159 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:39.083162 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:32:39.083188 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:39.085208 (+ 2020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:39.085701 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:39.085704 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:39.085706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:39.085782 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:39.086089 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:39.086092 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:39.086094 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:39.086308 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:39.087078 (+ 770us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:39.087167 (+ 89us) server_negotiation.cc:299] Negotiation successful
0926 07:32:39.087293 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 07:32:39.087630 3815 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.53.252" port: 39885 } has no permanent_uuid. Determining permanent_uuid...
I20260926 07:32:39.088758 3665 raft_consensus.cc:348] T 00000000000000000000000000000000 P d38b788c833747109549919f0de72509 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38b788c833747109549919f0de72509" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 35695 } } peers { permanent_uuid: "240849e070f34383a33ca7b0129d0874" member_type: VOTER last_known_addr { host: "127.22.53.253" port: 40619 } } peers { permanent_uuid: "56e96067b6d6486fa7ea34958450eb8b" member_type: VOTER last_known_addr { host: "127.22.53.252" port: 39885 } }
I20260926 07:32:39.089000 3665 raft_consensus.cc:374] T 00000000000000000000000000000000 P d38b788c833747109549919f0de72509 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:32:39.089066 3665 raft_consensus.cc:729] T 00000000000000000000000000000000 P d38b788c833747109549919f0de72509 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d38b788c833747109549919f0de72509, State: Initialized, Role: FOLLOWER
I20260926 07:32:39.089231 3668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:39.074850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60164 (local address 127.22.53.254:35695)
0926 07:32:39.075262 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:39.075266 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:39.075278 (+ 12us) server_negotiation.cc:407] Connection header received
0926 07:32:39.075315 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:39.075319 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:39.075370 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:39.075439 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:39.076280 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:39.077174 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:39.078364 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:39.078684 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:39.078819 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:39.082127 (+ 3308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:39.082131 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:39.082135 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:32:39.082167 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:39.084928 (+ 2761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:39.085901 (+ 973us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:39.085905 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:39.085907 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:39.085995 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:39.086590 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:39.086594 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:39.086596 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:39.087409 (+ 813us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:39.088751 (+ 1342us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:39.088817 (+ 66us) server_negotiation.cc:299] Negotiation successful
0926 07:32:39.088920 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334}
I20260926 07:32:39.089622 3665 consensus_queue.cc:260] T 00000000000000000000000000000000 P d38b788c833747109549919f0de72509 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38b788c833747109549919f0de72509" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 35695 } } peers { permanent_uuid: "240849e070f34383a33ca7b0129d0874" member_type: VOTER last_known_addr { host: "127.22.53.253" port: 40619 } } peers { permanent_uuid: "56e96067b6d6486fa7ea34958450eb8b" member_type: VOTER last_known_addr { host: "127.22.53.252" port: 39885 } }
I20260926 07:32:39.090178 3822 sys_catalog.cc:455] T 00000000000000000000000000000000 P d38b788c833747109549919f0de72509 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38b788c833747109549919f0de72509" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 35695 } } peers { permanent_uuid: "240849e070f34383a33ca7b0129d0874" member_type: VOTER last_known_addr { host: "127.22.53.253" port: 40619 } } peers { permanent_uuid: "56e96067b6d6486fa7ea34958450eb8b" member_type: VOTER last_known_addr { host: "127.22.53.252" port: 39885 } } }
I20260926 07:32:39.090441 3822 sys_catalog.cc:458] T 00000000000000000000000000000000 P d38b788c833747109549919f0de72509 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:32:39.090679 3740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 240849e070f34383a33ca7b0129d0874: Bootstrap starting.
I20260926 07:32:39.091112 3665 sys_catalog.cc:564] T 00000000000000000000000000000000 P d38b788c833747109549919f0de72509 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:32:39.093848 3740 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 240849e070f34383a33ca7b0129d0874: Neither blocks nor log segments found. Creating new log.
I20260926 07:32:39.094794 3740 log.cc:824] T 00000000000000000000000000000000 P 240849e070f34383a33ca7b0129d0874: Log is configured to *not* fsync() on all Append() calls
Sep 26 07:32:39 dist-test-slave-wdjc krb5kdc[3584](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.53.253@KRBTEST.COM
W20260926 07:32:39.096750 3833 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d38b788c833747109549919f0de72509: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:32:39.096870 3833 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:32:39.096202 3597 master_runner.cc:186] Error getting master registration for 127.22.53.253:40619: OK, instance_id { permanent_uuid: "240849e070f34383a33ca7b0129d0874" instance_seqno: 1790407958816800 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
Sep 26 07:32:39 dist-test-slave-wdjc krb5kdc[3584](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407959, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.252@KRBTEST.COM for kudu/127.22.53.252@KRBTEST.COM
I20260926 07:32:39.100306 3597 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 07:32:39.100508 3740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 240849e070f34383a33ca7b0129d0874: No bootstrap required, opened a new log
I20260926 07:32:39.101686 3741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:39.090519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36078 (local address 127.22.53.253:40619)
0926 07:32:39.090639 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:39.090644 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:39.090820 (+ 176us) server_negotiation.cc:407] Connection header received
0926 07:32:39.090991 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:39.090995 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:39.091066 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:39.091134 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:39.091834 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:39.092463 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:39.093520 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:39.093837 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:39.093941 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:39.097101 (+ 3160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:39.097105 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:39.097108 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:32:39.097133 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:39.098910 (+ 1777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:39.099449 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:39.099452 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:39.099454 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:39.099544 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:39.099876 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:39.099880 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:39.099881 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:39.100117 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:39.100322 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:39.101332 (+ 1010us) server_negotiation.cc:299] Negotiation successful
0926 07:32:39.101473 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 07:32:39.104234 3820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:39.088130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.252:39885 (local address 127.0.0.1:57336)
0926 07:32:39.088457 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:39.088473 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:32:39.088591 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:32:39.095578 (+ 6987us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:32:39.095581 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:32:39.095769 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:32:39.095885 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:39.095892 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:39.096957 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:39.096960 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:39.097787 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:39.097802 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:39.098430 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:39.098437 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:39.098538 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:39.099012 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:32:39.099030 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:32:39.100661 (+ 1631us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:32:39.102689 (+ 2028us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:32:39.102692 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:32:39.102694 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:32:39.102938 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:32:39.103238 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:32:39.103242 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:32:39.103244 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:32:39.103326 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:32:39.103730 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:32:39.103732 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:32:39.103830 (+ 98us) client_negotiation.cc:769] Sending connection context
0926 07:32:39.103915 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 07:32:39.104064 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254}
I20260926 07:32:39.104874 3818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:39.088348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57336 (local address 127.22.53.252:39885)
0926 07:32:39.095334 (+ 6986us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:39.095339 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:39.095356 (+ 17us) server_negotiation.cc:407] Connection header received
0926 07:32:39.095401 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:39.095404 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:39.095452 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:39.095995 (+ 543us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:39.096077 (+ 82us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:39.096840 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:39.097948 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:39.098284 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:39.098383 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:39.100826 (+ 2443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:39.100831 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:39.100834 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:32:39.100858 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:39.102583 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:39.103052 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:39.103055 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:39.103056 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:39.103144 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:39.103417 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:39.103419 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:39.103421 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:39.103633 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:39.104499 (+ 866us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:39.104595 (+ 96us) server_negotiation.cc:299] Negotiation successful
0926 07:32:39.104714 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6870}
I20260926 07:32:39.104813 3740 raft_consensus.cc:348] T 00000000000000000000000000000000 P 240849e070f34383a33ca7b0129d0874 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38b788c833747109549919f0de72509" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 35695 } } peers { permanent_uuid: "240849e070f34383a33ca7b0129d0874" member_type: VOTER last_known_addr { host: "127.22.53.253" port: 40619 } } peers { permanent_uuid: "56e96067b6d6486fa7ea34958450eb8b" member_type: VOTER last_known_addr { host: "127.22.53.252" port: 39885 } }
I20260926 07:32:39.105048 3740 raft_consensus.cc:374] T 00000000000000000000000000000000 P 240849e070f34383a33ca7b0129d0874 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:32:39.105099 3740 raft_consensus.cc:729] T 00000000000000000000000000000000 P 240849e070f34383a33ca7b0129d0874 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 240849e070f34383a33ca7b0129d0874, State: Initialized, Role: FOLLOWER
I20260926 07:32:39.105715 22743 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:32:39.105659 3740 consensus_queue.cc:260] T 00000000000000000000000000000000 P 240849e070f34383a33ca7b0129d0874 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38b788c833747109549919f0de72509" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 35695 } } peers { permanent_uuid: "240849e070f34383a33ca7b0129d0874" member_type: VOTER last_known_addr { host: "127.22.53.253" port: 40619 } } peers { permanent_uuid: "56e96067b6d6486fa7ea34958450eb8b" member_type: VOTER last_known_addr { host: "127.22.53.252" port: 39885 } }
I20260926 07:32:39.106246 3836 sys_catalog.cc:455] T 00000000000000000000000000000000 P 240849e070f34383a33ca7b0129d0874 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38b788c833747109549919f0de72509" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 35695 } } peers { permanent_uuid: "240849e070f34383a33ca7b0129d0874" member_type: VOTER last_known_addr { host: "127.22.53.253" port: 40619 } } peers { permanent_uuid: "56e96067b6d6486fa7ea34958450eb8b" member_type: VOTER last_known_addr { host: "127.22.53.252" port: 39885 } } }
I20260926 07:32:39.106470 3836 sys_catalog.cc:458] T 00000000000000000000000000000000 P 240849e070f34383a33ca7b0129d0874 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:32:39.107203 3740 sys_catalog.cc:564] T 00000000000000000000000000000000 P 240849e070f34383a33ca7b0129d0874 [sys.catalog]: configured and running, proceeding with master startup.
W20260926 07:32:39.113364 3854 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 240849e070f34383a33ca7b0129d0874: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:32:39.113562 3854 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:32:39.114907 3815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56e96067b6d6486fa7ea34958450eb8b: Bootstrap starting.
I20260926 07:32:39.119351 3815 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 56e96067b6d6486fa7ea34958450eb8b: Neither blocks nor log segments found. Creating new log.
I20260926 07:32:39.115828 3672 master_runner.cc:186] Error getting master registration for 127.22.53.252:39885: OK, instance_id { permanent_uuid: "56e96067b6d6486fa7ea34958450eb8b" instance_seqno: 1790407959045575 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 07:32:39.119839 3672 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 07:32:39.120337 3668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:39.109798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60174 (local address 127.22.53.254:35695)
0926 07:32:39.109969 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:39.109974 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:39.110481 (+ 507us) server_negotiation.cc:407] Connection header received
0926 07:32:39.110692 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:39.110696 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:39.110747 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:39.110807 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:39.111903 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:39.112735 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:39.114040 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:39.114404 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:39.114481 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:39.116464 (+ 1983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:39.116468 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:39.116471 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:32:39.116500 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:39.118313 (+ 1813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:39.118851 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:39.118855 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:39.118857 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:39.118943 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:39.119279 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:39.119284 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:39.119286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:39.119533 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:39.119647 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:39.120033 (+ 386us) server_negotiation.cc:299] Negotiation successful
0926 07:32:39.120171 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260926 07:32:39.120872 3815 log.cc:824] T 00000000000000000000000000000000 P 56e96067b6d6486fa7ea34958450eb8b: Log is configured to *not* fsync() on all Append() calls
I20260926 07:32:39.123862 3836 raft_consensus.cc:482] T 00000000000000000000000000000000 P 240849e070f34383a33ca7b0129d0874 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:32:39.124034 3836 raft_consensus.cc:504] T 00000000000000000000000000000000 P 240849e070f34383a33ca7b0129d0874 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38b788c833747109549919f0de72509" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 35695 } } peers { permanent_uuid: "240849e070f34383a33ca7b0129d0874" member_type: VOTER last_known_addr { host: "127.22.53.253" port: 40619 } } peers { permanent_uuid: "56e96067b6d6486fa7ea34958450eb8b" member_type: VOTER last_known_addr { host: "127.22.53.252" port: 39885 } }
I20260926 07:32:39.125159 3836 leader_election.cc:288] T 00000000000000000000000000000000 P 240849e070f34383a33ca7b0129d0874 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d38b788c833747109549919f0de72509 (127.22.53.254:35695), 56e96067b6d6486fa7ea34958450eb8b (127.22.53.252:39885)
I20260926 07:32:39.126165 3639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "240849e070f34383a33ca7b0129d0874" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d38b788c833747109549919f0de72509" is_pre_election: true
I20260926 07:32:39.126550 3639 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d38b788c833747109549919f0de72509 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 240849e070f34383a33ca7b0129d0874 in term 0.
I20260926 07:32:39.127345 3688 leader_election.cc:302] T 00000000000000000000000000000000 P 240849e070f34383a33ca7b0129d0874 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 240849e070f34383a33ca7b0129d0874, d38b788c833747109549919f0de72509; no voters:
I20260926 07:32:39.127723 3836 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 240849e070f34383a33ca7b0129d0874 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:32:39.127810 3836 raft_consensus.cc:482] T 00000000000000000000000000000000 P 240849e070f34383a33ca7b0129d0874 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:32:39.127856 3836 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 240849e070f34383a33ca7b0129d0874 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:32:39.129632 3836 raft_consensus.cc:504] T 00000000000000000000000000000000 P 240849e070f34383a33ca7b0129d0874 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38b788c833747109549919f0de72509" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 35695 } } peers { permanent_uuid: "240849e070f34383a33ca7b0129d0874" member_type: VOTER last_known_addr { host: "127.22.53.253" port: 40619 } } peers { permanent_uuid: "56e96067b6d6486fa7ea34958450eb8b" member_type: VOTER last_known_addr { host: "127.22.53.252" port: 39885 } }
I20260926 07:32:39.130582 3639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "240849e070f34383a33ca7b0129d0874" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d38b788c833747109549919f0de72509"
I20260926 07:32:39.130738 3639 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d38b788c833747109549919f0de72509 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:32:39.131779 3741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:39.108523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36090 (local address 127.22.53.253:40619)
0926 07:32:39.108670 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:39.108675 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:39.121637 (+ 12962us) server_negotiation.cc:407] Connection header received
0926 07:32:39.121837 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:39.121840 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:39.121893 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:39.121974 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:39.122826 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:39.123571 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:39.124844 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:39.125150 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:39.125234 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:39.127458 (+ 2224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:39.127462 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:39.127465 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:32:39.127494 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:39.129542 (+ 2048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:39.130246 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:39.130250 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:39.130251 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:39.130346 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:39.130671 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:39.130676 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:39.130680 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:39.130930 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:39.131054 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:39.131478 (+ 424us) server_negotiation.cc:299] Negotiation successful
0926 07:32:39.131606 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 07:32:39.132177 3639 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d38b788c833747109549919f0de72509 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 240849e070f34383a33ca7b0129d0874 in term 1.
I20260926 07:32:39.132826 3688 leader_election.cc:302] T 00000000000000000000000000000000 P 240849e070f34383a33ca7b0129d0874 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 240849e070f34383a33ca7b0129d0874, d38b788c833747109549919f0de72509; no voters:
I20260926 07:32:39.134704 3836 leader_election.cc:288] T 00000000000000000000000000000000 P 240849e070f34383a33ca7b0129d0874 [CANDIDATE]: Term 1 election: Requested vote from peers d38b788c833747109549919f0de72509 (127.22.53.254:35695), 56e96067b6d6486fa7ea34958450eb8b (127.22.53.252:39885)
I20260926 07:32:39.134788 3815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56e96067b6d6486fa7ea34958450eb8b: No bootstrap required, opened a new log
I20260926 07:32:39.134919 3836 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 240849e070f34383a33ca7b0129d0874 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:32:39.135057 3836 raft_consensus.cc:686] T 00000000000000000000000000000000 P 240849e070f34383a33ca7b0129d0874 [term 1 LEADER]: Becoming Leader. State: Replica: 240849e070f34383a33ca7b0129d0874, State: Running, Role: LEADER
I20260926 07:32:39.135722 3836 consensus_queue.cc:237] T 00000000000000000000000000000000 P 240849e070f34383a33ca7b0129d0874 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38b788c833747109549919f0de72509" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 35695 } } peers { permanent_uuid: "240849e070f34383a33ca7b0129d0874" member_type: VOTER last_known_addr { host: "127.22.53.253" port: 40619 } } peers { permanent_uuid: "56e96067b6d6486fa7ea34958450eb8b" member_type: VOTER last_known_addr { host: "127.22.53.252" port: 39885 } }
I20260926 07:32:39.139460 3836 sys_catalog.cc:455] T 00000000000000000000000000000000 P 240849e070f34383a33ca7b0129d0874 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 240849e070f34383a33ca7b0129d0874. Latest consensus state: current_term: 1 leader_uuid: "240849e070f34383a33ca7b0129d0874" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38b788c833747109549919f0de72509" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 35695 } } peers { permanent_uuid: "240849e070f34383a33ca7b0129d0874" member_type: VOTER last_known_addr { host: "127.22.53.253" port: 40619 } } peers { permanent_uuid: "56e96067b6d6486fa7ea34958450eb8b" member_type: VOTER last_known_addr { host: "127.22.53.252" port: 39885 } } }
I20260926 07:32:39.139417 3815 raft_consensus.cc:348] T 00000000000000000000000000000000 P 56e96067b6d6486fa7ea34958450eb8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38b788c833747109549919f0de72509" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 35695 } } peers { permanent_uuid: "240849e070f34383a33ca7b0129d0874" member_type: VOTER last_known_addr { host: "127.22.53.253" port: 40619 } } peers { permanent_uuid: "56e96067b6d6486fa7ea34958450eb8b" member_type: VOTER last_known_addr { host: "127.22.53.252" port: 39885 } }
I20260926 07:32:39.139696 3815 raft_consensus.cc:374] T 00000000000000000000000000000000 P 56e96067b6d6486fa7ea34958450eb8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:32:39.139762 3815 raft_consensus.cc:729] T 00000000000000000000000000000000 P 56e96067b6d6486fa7ea34958450eb8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e96067b6d6486fa7ea34958450eb8b, State: Initialized, Role: FOLLOWER
I20260926 07:32:39.139741 3836 sys_catalog.cc:458] T 00000000000000000000000000000000 P 240849e070f34383a33ca7b0129d0874 [sys.catalog]: This master's current role is: LEADER
I20260926 07:32:39.140287 3815 consensus_queue.cc:260] T 00000000000000000000000000000000 P 56e96067b6d6486fa7ea34958450eb8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38b788c833747109549919f0de72509" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 35695 } } peers { permanent_uuid: "240849e070f34383a33ca7b0129d0874" member_type: VOTER last_known_addr { host: "127.22.53.253" port: 40619 } } peers { permanent_uuid: "56e96067b6d6486fa7ea34958450eb8b" member_type: VOTER last_known_addr { host: "127.22.53.252" port: 39885 } }
I20260926 07:32:39.142254 3815 sys_catalog.cc:564] T 00000000000000000000000000000000 P 56e96067b6d6486fa7ea34958450eb8b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:32:39.142300 3789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "240849e070f34383a33ca7b0129d0874" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e96067b6d6486fa7ea34958450eb8b" is_pre_election: true
I20260926 07:32:39.142611 3789 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 56e96067b6d6486fa7ea34958450eb8b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 240849e070f34383a33ca7b0129d0874 in term 0.
I20260926 07:32:39.142894 3788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "240849e070f34383a33ca7b0129d0874" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e96067b6d6486fa7ea34958450eb8b"
I20260926 07:32:39.143010 3788 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 56e96067b6d6486fa7ea34958450eb8b [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:32:39.144683 3788 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 56e96067b6d6486fa7ea34958450eb8b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 240849e070f34383a33ca7b0129d0874 in term 1.
I20260926 07:32:39.146641 3859 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:32:39.147830 3859 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:32:39.151911 3860 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56e96067b6d6486fa7ea34958450eb8b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38b788c833747109549919f0de72509" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 35695 } } peers { permanent_uuid: "240849e070f34383a33ca7b0129d0874" member_type: VOTER last_known_addr { host: "127.22.53.253" port: 40619 } } peers { permanent_uuid: "56e96067b6d6486fa7ea34958450eb8b" member_type: VOTER last_known_addr { host: "127.22.53.252" port: 39885 } } }
I20260926 07:32:39.152186 3860 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56e96067b6d6486fa7ea34958450eb8b [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:32:39.157419 3788 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 56e96067b6d6486fa7ea34958450eb8b [term 1 FOLLOWER]: Refusing update from remote peer 240849e070f34383a33ca7b0129d0874: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:32:39.157850 3818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:39.113255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57338 (local address 127.22.53.252:39885)
0926 07:32:39.113396 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:39.113402 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:39.121276 (+ 7874us) server_negotiation.cc:407] Connection header received
0926 07:32:39.121334 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:39.121339 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:39.121395 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:39.121485 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:39.122747 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:39.123573 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:39.145729 (+ 22156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:39.146072 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:39.146183 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:39.154038 (+ 7855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:39.154042 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:39.154045 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:32:39.154075 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:39.155819 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:39.156360 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:39.156364 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:39.156365 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:39.156454 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:39.156801 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:39.156805 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:39.156809 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:39.157052 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:39.157187 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:39.157569 (+ 382us) server_negotiation.cc:299] Negotiation successful
0926 07:32:39.157699 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 07:32:39.158700 3836 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 240849e070f34383a33ca7b0129d0874 [LEADER]: Connected to new peer: Peer: permanent_uuid: "56e96067b6d6486fa7ea34958450eb8b" member_type: VOTER last_known_addr { host: "127.22.53.252" port: 39885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:32:39.163409 3860 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56e96067b6d6486fa7ea34958450eb8b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 240849e070f34383a33ca7b0129d0874. Latest consensus state: current_term: 1 leader_uuid: "240849e070f34383a33ca7b0129d0874" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38b788c833747109549919f0de72509" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 35695 } } peers { permanent_uuid: "240849e070f34383a33ca7b0129d0874" member_type: VOTER last_known_addr { host: "127.22.53.253" port: 40619 } } peers { permanent_uuid: "56e96067b6d6486fa7ea34958450eb8b" member_type: VOTER last_known_addr { host: "127.22.53.252" port: 39885 } } }
I20260926 07:32:39.163754 3860 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56e96067b6d6486fa7ea34958450eb8b [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:32:39.165045 3639 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d38b788c833747109549919f0de72509 [term 1 FOLLOWER]: Refusing update from remote peer 240849e070f34383a33ca7b0129d0874: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:32:39.165755 3857 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 240849e070f34383a33ca7b0129d0874 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d38b788c833747109549919f0de72509" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 35695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 07:32:39.166565 3871 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 56e96067b6d6486fa7ea34958450eb8b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:32:39.168399 3871 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:32:39.169804 3822 sys_catalog.cc:455] T 00000000000000000000000000000000 P d38b788c833747109549919f0de72509 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 240849e070f34383a33ca7b0129d0874. Latest consensus state: current_term: 1 leader_uuid: "240849e070f34383a33ca7b0129d0874" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38b788c833747109549919f0de72509" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 35695 } } peers { permanent_uuid: "240849e070f34383a33ca7b0129d0874" member_type: VOTER last_known_addr { host: "127.22.53.253" port: 40619 } } peers { permanent_uuid: "56e96067b6d6486fa7ea34958450eb8b" member_type: VOTER last_known_addr { host: "127.22.53.252" port: 39885 } } }
I20260926 07:32:39.169942 3822 sys_catalog.cc:458] T 00000000000000000000000000000000 P d38b788c833747109549919f0de72509 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:32:39.173615 3872 mvcc.cc:205] Tried to move back new op lower bound from 7333511000699613184 to 7333511000626601984. Current Snapshot: MvccSnapshot[applied={T|T < 7333511000699613184}]
I20260926 07:32:39.174916 3860 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56e96067b6d6486fa7ea34958450eb8b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "240849e070f34383a33ca7b0129d0874" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38b788c833747109549919f0de72509" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 35695 } } peers { permanent_uuid: "240849e070f34383a33ca7b0129d0874" member_type: VOTER last_known_addr { host: "127.22.53.253" port: 40619 } } peers { permanent_uuid: "56e96067b6d6486fa7ea34958450eb8b" member_type: VOTER last_known_addr { host: "127.22.53.252" port: 39885 } } }
I20260926 07:32:39.175053 3860 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56e96067b6d6486fa7ea34958450eb8b [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:32:39.175452 3836 sys_catalog.cc:455] T 00000000000000000000000000000000 P 240849e070f34383a33ca7b0129d0874 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "240849e070f34383a33ca7b0129d0874" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38b788c833747109549919f0de72509" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 35695 } } peers { permanent_uuid: "240849e070f34383a33ca7b0129d0874" member_type: VOTER last_known_addr { host: "127.22.53.253" port: 40619 } } peers { permanent_uuid: "56e96067b6d6486fa7ea34958450eb8b" member_type: VOTER last_known_addr { host: "127.22.53.252" port: 39885 } } }
I20260926 07:32:39.175611 3836 sys_catalog.cc:458] T 00000000000000000000000000000000 P 240849e070f34383a33ca7b0129d0874 [sys.catalog]: This master's current role is: LEADER
I20260926 07:32:39.176651 3836 sys_catalog.cc:455] T 00000000000000000000000000000000 P 240849e070f34383a33ca7b0129d0874 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "240849e070f34383a33ca7b0129d0874" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38b788c833747109549919f0de72509" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 35695 } } peers { permanent_uuid: "240849e070f34383a33ca7b0129d0874" member_type: VOTER last_known_addr { host: "127.22.53.253" port: 40619 } } peers { permanent_uuid: "56e96067b6d6486fa7ea34958450eb8b" member_type: VOTER last_known_addr { host: "127.22.53.252" port: 39885 } } }
I20260926 07:32:39.176785 3836 sys_catalog.cc:458] T 00000000000000000000000000000000 P 240849e070f34383a33ca7b0129d0874 [sys.catalog]: This master's current role is: LEADER
I20260926 07:32:39.177513 3822 sys_catalog.cc:455] T 00000000000000000000000000000000 P d38b788c833747109549919f0de72509 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "240849e070f34383a33ca7b0129d0874" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38b788c833747109549919f0de72509" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 35695 } } peers { permanent_uuid: "240849e070f34383a33ca7b0129d0874" member_type: VOTER last_known_addr { host: "127.22.53.253" port: 40619 } } peers { permanent_uuid: "56e96067b6d6486fa7ea34958450eb8b" member_type: VOTER last_known_addr { host: "127.22.53.252" port: 39885 } } }
I20260926 07:32:39.177604 3822 sys_catalog.cc:458] T 00000000000000000000000000000000 P d38b788c833747109549919f0de72509 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:32:39.178015 3876 mvcc.cc:205] Tried to move back new op lower bound from 7333511000699613184 to 7333511000626601984. Current Snapshot: MvccSnapshot[applied={T|T < 7333511000699613184}]
I20260926 07:32:39.183082 3859 catalog_manager.cc:1347] Generated new cluster ID: c5cd105b173c4145857e9e97039237cd
I20260926 07:32:39.183174 3859 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:32:39.213073 3859 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:32:39.214597 3859 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:32:39.236735 3859 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 240849e070f34383a33ca7b0129d0874: Generated new TSK 0
I20260926 07:32:39.237841 3859 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:32:40.098372 3833 catalog_manager.cc:1259] Loaded cluster ID: c5cd105b173c4145857e9e97039237cd
I20260926 07:32:40.098508 3833 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d38b788c833747109549919f0de72509: loading cluster ID for follower catalog manager: success
I20260926 07:32:40.100315 3833 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d38b788c833747109549919f0de72509: acquiring CA information for follower catalog manager: success
I20260926 07:32:40.101770 3833 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d38b788c833747109549919f0de72509: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 07:32:40.169862 3871 catalog_manager.cc:1259] Loaded cluster ID: c5cd105b173c4145857e9e97039237cd
I20260926 07:32:40.169965 3871 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 56e96067b6d6486fa7ea34958450eb8b: loading cluster ID for follower catalog manager: success
I20260926 07:32:40.171742 3871 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 56e96067b6d6486fa7ea34958450eb8b: acquiring CA information for follower catalog manager: success
I20260926 07:32:40.173094 3871 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 56e96067b6d6486fa7ea34958450eb8b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 07:32:40.309782 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 3597
I20260926 07:32:40.317353 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 3672
I20260926 07:32:40.324162 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 3746
2026-09-26T07:32:40Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3186 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407783529717-22743-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:32:40 dist-test-slave-wdjc krb5kdc[3889](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:32:40 dist-test-slave-wdjc krb5kdc[3889](info): set up 2 sockets
Sep 26 07:32:40 dist-test-slave-wdjc krb5kdc[3889](info): commencing operation
krb5kdc: starting...
W20260926 07:32:41.381875 22743 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407783529717-22743-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407783529717-22743-0/krb5kdc/test-user.keytab.
Sep 26 07:32:41 dist-test-slave-wdjc krb5kdc[3889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407961, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:32:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:32:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.53.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.53.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.53.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab.
Entry for principal kudu/127.22.53.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab.
I20260926 07:32:41.517058 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407783529717-22743-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407783529717-22743-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407783529717-22743-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.53.254:37077
--webserver_interface=127.22.53.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab
--principal=kudu/127.22.53.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:43211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.53.254:37077
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:32:41.635694 3902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:32:41.636039 3902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:32:41.636124 3902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:32:41.640488 3902 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:32:41.640626 3902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:32:41.640650 3902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:32:41.640671 3902 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:32:41.640740 3902 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:32:41.647243 3902 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:43211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407783529717-22743-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407783529717-22743-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.53.254:37077
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab
--principal=kudu/127.22.53.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.53.254:37077
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.53.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3902
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407783529717-22743-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:32:41.648469 3902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:32:41.649686 3902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 07:32:41.660403 3902 server_base.cc:1034] running on GCE node
W20260926 07:32:41.660300 3910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:41.660300 3907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:41.660300 3908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:32:41.661217 3902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:32:41.662457 3902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:32:41.663653 3902 hybrid_clock.cc:648] HybridClock initialized: now 1790407961663625 us; error 49 us; skew 500 ppm
Sep 26 07:32:41 dist-test-slave-wdjc krb5kdc[3889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407961, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:32:41.666914 3902 init.cc:377] Logged in from keytab as kudu/127.22.53.254@KRBTEST.COM (short username kudu)
I20260926 07:32:41.667989 3902 webserver.cc:460] Webserver started at http://127.22.53.254:37127/ using document root <none> and password file <none>
I20260926 07:32:41.668675 3902 fs_manager.cc:359] Metadata directory not provided
I20260926 07:32:41.668772 3902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:32:41.669014 3902 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:32:41.671284 3902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407783529717-22743-0/minicluster-data/master-0/data/instance:
uuid: "62df657ca940409e8188194beb4196a8"
format_stamp: "Formatted at 2026-09-26 07:32:41 on dist-test-slave-wdjc"
server_key: "7fdb059ccaaaba8d1cd1a9f4098dfb89"
server_key_iv: "39c2d941c967b83c5e920745eca71079"
server_key_version: "encryptionkey@0"
I20260926 07:32:41.672487 3902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407783529717-22743-0/minicluster-data/master-0/wal/instance:
uuid: "62df657ca940409e8188194beb4196a8"
format_stamp: "Formatted at 2026-09-26 07:32:41 on dist-test-slave-wdjc"
server_key: "7fdb059ccaaaba8d1cd1a9f4098dfb89"
server_key_iv: "39c2d941c967b83c5e920745eca71079"
server_key_version: "encryptionkey@0"
I20260926 07:32:41.678134 3902 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 07:32:41.681340 3917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:41.682797 3902 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 07:32:41.682974 3902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407783529717-22743-0/minicluster-data/master-0/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407783529717-22743-0/minicluster-data/master-0/wal
uuid: "62df657ca940409e8188194beb4196a8"
format_stamp: "Formatted at 2026-09-26 07:32:41 on dist-test-slave-wdjc"
server_key: "7fdb059ccaaaba8d1cd1a9f4098dfb89"
server_key_iv: "39c2d941c967b83c5e920745eca71079"
server_key_version: "encryptionkey@0"
I20260926 07:32:41.683123 3902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407783529717-22743-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407783529717-22743-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407783529717-22743-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:32:41.714540 3902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:32:41.719022 3902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:32:41.719300 3902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:32:41.727734 3902 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.254:37077
I20260926 07:32:41.727756 3969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.254:37077 every 8 connection(s)
I20260926 07:32:41.728991 3902 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
I20260926 07:32:41.732511 3970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:32:41.735288 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 3902
I20260926 07:32:41.735443 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407783529717-22743-0/minicluster-data/master-0/wal/instance
I20260926 07:32:41.735754 22743 external_mini_cluster.cc:1428] Setting key 55f12fb6e08090a736fb83de23a7d1a3
I20260926 07:32:41.740926 3970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62df657ca940409e8188194beb4196a8: Bootstrap starting.
I20260926 07:32:41.743985 3970 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 62df657ca940409e8188194beb4196a8: Neither blocks nor log segments found. Creating new log.
I20260926 07:32:41.745049 3970 log.cc:824] T 00000000000000000000000000000000 P 62df657ca940409e8188194beb4196a8: Log is configured to *not* fsync() on all Append() calls
Sep 26 07:32:41 dist-test-slave-wdjc krb5kdc[3889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407961, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.53.254@KRBTEST.COM
I20260926 07:32:41.747884 3970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62df657ca940409e8188194beb4196a8: No bootstrap required, opened a new log
I20260926 07:32:41.752252 3970 raft_consensus.cc:348] T 00000000000000000000000000000000 P 62df657ca940409e8188194beb4196a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62df657ca940409e8188194beb4196a8" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 37077 } }
I20260926 07:32:41.752612 3970 raft_consensus.cc:374] T 00000000000000000000000000000000 P 62df657ca940409e8188194beb4196a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:32:41.752686 3970 raft_consensus.cc:729] T 00000000000000000000000000000000 P 62df657ca940409e8188194beb4196a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62df657ca940409e8188194beb4196a8, State: Initialized, Role: FOLLOWER
I20260926 07:32:41.753147 3973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:41.737381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56448 (local address 127.22.53.254:37077)
0926 07:32:41.737864 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:41.737888 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:41.737928 (+ 40us) server_negotiation.cc:407] Connection header received
0926 07:32:41.738597 (+ 669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:41.738617 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:41.738988 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:41.739310 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:41.740322 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:41.741614 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:41.743275 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:41.744021 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:41.744278 (+ 257us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:41.747251 (+ 2973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:41.747257 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:41.747273 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:32:41.747307 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:41.749666 (+ 2359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:41.750228 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:41.750234 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:41.750239 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:41.750367 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:41.750668 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:41.750671 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:41.750673 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:41.751145 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:41.751410 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:41.751705 (+ 295us) server_negotiation.cc:299] Negotiation successful
0926 07:32:41.752039 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":173,"threads_started":1}
I20260926 07:32:41.753255 3970 consensus_queue.cc:260] T 00000000000000000000000000000000 P 62df657ca940409e8188194beb4196a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62df657ca940409e8188194beb4196a8" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 37077 } }
I20260926 07:32:41.753472 3970 raft_consensus.cc:388] T 00000000000000000000000000000000 P 62df657ca940409e8188194beb4196a8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:32:41.753556 3970 raft_consensus.cc:482] T 00000000000000000000000000000000 P 62df657ca940409e8188194beb4196a8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:32:41.753672 3970 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 62df657ca940409e8188194beb4196a8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:32:41.755069 3970 raft_consensus.cc:504] T 00000000000000000000000000000000 P 62df657ca940409e8188194beb4196a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62df657ca940409e8188194beb4196a8" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 37077 } }
I20260926 07:32:41.755518 3970 leader_election.cc:302] T 00000000000000000000000000000000 P 62df657ca940409e8188194beb4196a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 62df657ca940409e8188194beb4196a8; no voters:
I20260926 07:32:41.755978 3970 leader_election.cc:288] T 00000000000000000000000000000000 P 62df657ca940409e8188194beb4196a8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:32:41.756069 3975 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 62df657ca940409e8188194beb4196a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:32:41.756284 3975 raft_consensus.cc:686] T 00000000000000000000000000000000 P 62df657ca940409e8188194beb4196a8 [term 1 LEADER]: Becoming Leader. State: Replica: 62df657ca940409e8188194beb4196a8, State: Running, Role: LEADER
I20260926 07:32:41.756655 3975 consensus_queue.cc:237] T 00000000000000000000000000000000 P 62df657ca940409e8188194beb4196a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62df657ca940409e8188194beb4196a8" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 37077 } }
I20260926 07:32:41.757279 22743 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:32:41.757586 3970 sys_catalog.cc:564] T 00000000000000000000000000000000 P 62df657ca940409e8188194beb4196a8 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:32:41.759080 3976 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62df657ca940409e8188194beb4196a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62df657ca940409e8188194beb4196a8. Latest consensus state: current_term: 1 leader_uuid: "62df657ca940409e8188194beb4196a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62df657ca940409e8188194beb4196a8" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 37077 } } }
I20260926 07:32:41.759254 3976 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62df657ca940409e8188194beb4196a8 [sys.catalog]: This master's current role is: LEADER
I20260926 07:32:41.759747 3977 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62df657ca940409e8188194beb4196a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "62df657ca940409e8188194beb4196a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62df657ca940409e8188194beb4196a8" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 37077 } } }
I20260926 07:32:41.760581 3977 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62df657ca940409e8188194beb4196a8 [sys.catalog]: This master's current role is: LEADER
I20260926 07:32:41.760716 3984 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:32:41.763845 3984 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:32:41.771425 3984 catalog_manager.cc:1347] Generated new cluster ID: 428122a775084919a2acbb1db67c76b5
I20260926 07:32:41.771538 3984 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:32:41.783172 3984 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:32:41.784313 3984 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:32:41.804462 3984 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 62df657ca940409e8188194beb4196a8: Generated new TSK 0
I20260926 07:32:41.805370 3984 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:32:41.815106 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 3902
2026-09-26T07:32:41Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1486 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:32:41 dist-test-slave-wdjc krb5kdc[3997](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:32:41 dist-test-slave-wdjc krb5kdc[3997](info): set up 2 sockets
Sep 26 07:32:41 dist-test-slave-wdjc krb5kdc[3997](info): commencing operation
krb5kdc: starting...
W20260926 07:32:42.860976 22743 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/krb5kdc/test-user.keytab.
Sep 26 07:32:42 dist-test-slave-wdjc krb5kdc[3997](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407962, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:32:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:32:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.53.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.53.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.53.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab.
Entry for principal kudu/127.22.53.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab.
I20260926 07:32:42.992122 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.53.254:35543
--webserver_interface=127.22.53.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab
--principal=kudu/127.22.53.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:46315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.53.254:35543
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:32:43.111642 4010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:32:43.112027 4010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:32:43.112110 4010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:32:43.116447 4010 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:32:43.116618 4010 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:32:43.116657 4010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:32:43.116717 4010 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:32:43.116775 4010 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:32:43.122730 4010 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:46315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.53.254:35543
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab
--principal=kudu/127.22.53.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.53.254:35543
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.53.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4010
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:32:43.124042 4010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:32:43.125252 4010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:32:43.134533 4015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:43.134613 4018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:32:43.134608 4010 server_base.cc:1034] running on GCE node
W20260926 07:32:43.134531 4016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:32:43.135459 4010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:32:43.136847 4010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:32:43.138072 4010 hybrid_clock.cc:648] HybridClock initialized: now 1790407963138056 us; error 38 us; skew 500 ppm
Sep 26 07:32:43 dist-test-slave-wdjc krb5kdc[3997](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407963, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:32:43.146428 4010 init.cc:377] Logged in from keytab as kudu/127.22.53.254@KRBTEST.COM (short username kudu)
I20260926 07:32:43.147625 4010 webserver.cc:460] Webserver started at http://127.22.53.254:37215/ using document root <none> and password file <none>
I20260926 07:32:43.148276 4010 fs_manager.cc:359] Metadata directory not provided
I20260926 07:32:43.148384 4010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:32:43.148710 4010 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:32:43.151031 4010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/master-0/data/instance:
uuid: "095dfa56bdf344f6b50d5cc18be740cd"
format_stamp: "Formatted at 2026-09-26 07:32:43 on dist-test-slave-wdjc"
server_key: "fa15f9a965a45d0401f978b68d44e8d6"
server_key_iv: "85062671625e999fae5efde875b2c3d0"
server_key_version: "encryptionkey@0"
I20260926 07:32:43.151719 4010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/master-0/wal/instance:
uuid: "095dfa56bdf344f6b50d5cc18be740cd"
format_stamp: "Formatted at 2026-09-26 07:32:43 on dist-test-slave-wdjc"
server_key: "fa15f9a965a45d0401f978b68d44e8d6"
server_key_iv: "85062671625e999fae5efde875b2c3d0"
server_key_version: "encryptionkey@0"
I20260926 07:32:43.156661 4010 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 07:32:43.159637 4025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:43.161242 4010 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:32:43.161432 4010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/master-0/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/master-0/wal
uuid: "095dfa56bdf344f6b50d5cc18be740cd"
format_stamp: "Formatted at 2026-09-26 07:32:43 on dist-test-slave-wdjc"
server_key: "fa15f9a965a45d0401f978b68d44e8d6"
server_key_iv: "85062671625e999fae5efde875b2c3d0"
server_key_version: "encryptionkey@0"
I20260926 07:32:43.161581 4010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:32:43.178471 4010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:32:43.182400 4010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:32:43.182626 4010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:32:43.190748 4010 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.254:35543
I20260926 07:32:43.191085 4077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.254:35543 every 8 connection(s)
I20260926 07:32:43.191948 4010 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
I20260926 07:32:43.195165 4078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:32:43.200410 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 4010
I20260926 07:32:43.200594 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/master-0/wal/instance
I20260926 07:32:43.200883 22743 external_mini_cluster.cc:1428] Setting key d03fd3834f8e772e2bd3529ca76ec2fc
I20260926 07:32:43.202471 4078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 095dfa56bdf344f6b50d5cc18be740cd: Bootstrap starting.
I20260926 07:32:43.205075 4078 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 095dfa56bdf344f6b50d5cc18be740cd: Neither blocks nor log segments found. Creating new log.
I20260926 07:32:43.205909 4078 log.cc:824] T 00000000000000000000000000000000 P 095dfa56bdf344f6b50d5cc18be740cd: Log is configured to *not* fsync() on all Append() calls
I20260926 07:32:43.208024 4078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 095dfa56bdf344f6b50d5cc18be740cd: No bootstrap required, opened a new log
Sep 26 07:32:43 dist-test-slave-wdjc krb5kdc[3997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407962, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.53.254@KRBTEST.COM
I20260926 07:32:43.213366 4078 raft_consensus.cc:348] T 00000000000000000000000000000000 P 095dfa56bdf344f6b50d5cc18be740cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "095dfa56bdf344f6b50d5cc18be740cd" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 35543 } }
I20260926 07:32:43.213752 4078 raft_consensus.cc:374] T 00000000000000000000000000000000 P 095dfa56bdf344f6b50d5cc18be740cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:32:43.213819 4078 raft_consensus.cc:729] T 00000000000000000000000000000000 P 095dfa56bdf344f6b50d5cc18be740cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 095dfa56bdf344f6b50d5cc18be740cd, State: Initialized, Role: FOLLOWER
I20260926 07:32:43.214375 4078 consensus_queue.cc:260] T 00000000000000000000000000000000 P 095dfa56bdf344f6b50d5cc18be740cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "095dfa56bdf344f6b50d5cc18be740cd" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 35543 } }
I20260926 07:32:43.214570 4078 raft_consensus.cc:388] T 00000000000000000000000000000000 P 095dfa56bdf344f6b50d5cc18be740cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:32:43.214646 4078 raft_consensus.cc:482] T 00000000000000000000000000000000 P 095dfa56bdf344f6b50d5cc18be740cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:32:43.214772 4078 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 095dfa56bdf344f6b50d5cc18be740cd [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:32:43.215785 4078 raft_consensus.cc:504] T 00000000000000000000000000000000 P 095dfa56bdf344f6b50d5cc18be740cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "095dfa56bdf344f6b50d5cc18be740cd" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 35543 } }
I20260926 07:32:43.216248 4078 leader_election.cc:302] T 00000000000000000000000000000000 P 095dfa56bdf344f6b50d5cc18be740cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 095dfa56bdf344f6b50d5cc18be740cd; no voters:
I20260926 07:32:43.216676 4078 leader_election.cc:288] T 00000000000000000000000000000000 P 095dfa56bdf344f6b50d5cc18be740cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:32:43.216794 4083 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 095dfa56bdf344f6b50d5cc18be740cd [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:32:43.217022 4083 raft_consensus.cc:686] T 00000000000000000000000000000000 P 095dfa56bdf344f6b50d5cc18be740cd [term 1 LEADER]: Becoming Leader. State: Replica: 095dfa56bdf344f6b50d5cc18be740cd, State: Running, Role: LEADER
I20260926 07:32:43.217154 4081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:43.202538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40142 (local address 127.22.53.254:35543)
0926 07:32:43.203051 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:43.203068 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:43.203108 (+ 40us) server_negotiation.cc:407] Connection header received
0926 07:32:43.203851 (+ 743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:43.203870 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:43.204259 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:43.204694 (+ 435us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:43.205584 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:43.206694 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:43.207599 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:43.208095 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:43.208273 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:43.211183 (+ 2910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:43.211192 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:43.211232 (+ 40us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:32:43.211281 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:43.213451 (+ 2170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:43.214130 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:43.214136 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:43.214142 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:43.214307 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:43.214616 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:43.214626 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:43.214627 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:43.215125 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:43.215403 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:43.215701 (+ 298us) server_negotiation.cc:299] Negotiation successful
0926 07:32:43.216067 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":161,"threads_started":1}
I20260926 07:32:43.217414 4083 consensus_queue.cc:237] T 00000000000000000000000000000000 P 095dfa56bdf344f6b50d5cc18be740cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "095dfa56bdf344f6b50d5cc18be740cd" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 35543 } }
I20260926 07:32:43.218106 4078 sys_catalog.cc:564] T 00000000000000000000000000000000 P 095dfa56bdf344f6b50d5cc18be740cd [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:32:43.219153 4084 sys_catalog.cc:455] T 00000000000000000000000000000000 P 095dfa56bdf344f6b50d5cc18be740cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "095dfa56bdf344f6b50d5cc18be740cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "095dfa56bdf344f6b50d5cc18be740cd" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 35543 } } }
I20260926 07:32:43.219385 4084 sys_catalog.cc:458] T 00000000000000000000000000000000 P 095dfa56bdf344f6b50d5cc18be740cd [sys.catalog]: This master's current role is: LEADER
I20260926 07:32:43.220158 4085 sys_catalog.cc:455] T 00000000000000000000000000000000 P 095dfa56bdf344f6b50d5cc18be740cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 095dfa56bdf344f6b50d5cc18be740cd. Latest consensus state: current_term: 1 leader_uuid: "095dfa56bdf344f6b50d5cc18be740cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "095dfa56bdf344f6b50d5cc18be740cd" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 35543 } } }
I20260926 07:32:43.220296 4085 sys_catalog.cc:458] T 00000000000000000000000000000000 P 095dfa56bdf344f6b50d5cc18be740cd [sys.catalog]: This master's current role is: LEADER
W20260926 07:32:43.223647 4098 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 095dfa56bdf344f6b50d5cc18be740cd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:32:43.223789 4098 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:32:43.223945 4092 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:32:43.225005 4092 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:32:43.232016 4092 catalog_manager.cc:1347] Generated new cluster ID: 96350a859a0d476b9eb3fc198c396466
I20260926 07:32:43.232134 4092 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:32:43.245033 4092 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:32:43.246152 4092 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:32:43.255416 4092 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 095dfa56bdf344f6b50d5cc18be740cd: Generated new TSK 0
I20260926 07:32:43.256282 4092 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.53.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.53.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.53.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.193.keytab.
Entry for principal kudu/127.22.53.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.193.keytab.
I20260926 07:32:43.293128 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.53.193:0
--local_ip_for_outbound_sockets=127.22.53.193
--webserver_interface=127.22.53.193
--webserver_port=0
--tserver_master_addrs=127.22.53.254:35543
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.193.keytab
--principal=kudu/127.22.53.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:46315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:32:43.442178 4104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:32:43.442533 4104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:32:43.442624 4104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:32:43.446981 4104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:32:43.447115 4104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:32:43.447218 4104 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.53.193
I20260926 07:32:43.453310 4104 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:46315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.193.keytab
--principal=kudu/127.22.53.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.53.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.53.193
--webserver_port=0
--tserver_master_addrs=127.22.53.254:35543
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4104
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.53.193
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:32:43.454568 4104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:32:43.455693 4104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 07:32:43.466933 4104 server_base.cc:1034] running on GCE node
W20260926 07:32:43.466861 4109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:43.466866 4110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:43.466861 4112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:32:43.467645 4104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:32:43.468358 4104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:32:43.469542 4104 hybrid_clock.cc:648] HybridClock initialized: now 1790407963469519 us; error 40 us; skew 500 ppm
Sep 26 07:32:43 dist-test-slave-wdjc krb5kdc[3997](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407963, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:32:43.472908 4104 init.cc:377] Logged in from keytab as kudu/127.22.53.193@KRBTEST.COM (short username kudu)
I20260926 07:32:43.474020 4104 webserver.cc:460] Webserver started at http://127.22.53.193:46039/ using document root <none> and password file <none>
I20260926 07:32:43.474696 4104 fs_manager.cc:359] Metadata directory not provided
I20260926 07:32:43.474772 4104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:32:43.475020 4104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:32:43.477556 4104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-0/data/instance:
uuid: "84670aa208b740d1bdde3276c1a33264"
format_stamp: "Formatted at 2026-09-26 07:32:43 on dist-test-slave-wdjc"
server_key: "5033f0687a2c951f94f1887fff755d26"
server_key_iv: "9c598874f2d5cacc1ad23cfdada1bf43"
server_key_version: "encryptionkey@0"
I20260926 07:32:43.478256 4104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-0/wal/instance:
uuid: "84670aa208b740d1bdde3276c1a33264"
format_stamp: "Formatted at 2026-09-26 07:32:43 on dist-test-slave-wdjc"
server_key: "5033f0687a2c951f94f1887fff755d26"
server_key_iv: "9c598874f2d5cacc1ad23cfdada1bf43"
server_key_version: "encryptionkey@0"
I20260926 07:32:43.483098 4104 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260926 07:32:43.486340 4119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:43.487979 4104 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 07:32:43.488191 4104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-0/wal
uuid: "84670aa208b740d1bdde3276c1a33264"
format_stamp: "Formatted at 2026-09-26 07:32:43 on dist-test-slave-wdjc"
server_key: "5033f0687a2c951f94f1887fff755d26"
server_key_iv: "9c598874f2d5cacc1ad23cfdada1bf43"
server_key_version: "encryptionkey@0"
I20260926 07:32:43.488389 4104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:32:43.514791 4104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:32:43.518738 4104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:32:43.519037 4104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:32:43.519758 4104 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:32:43.520828 4104 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:32:43.520903 4104 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:43.520980 4104 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:32:43.521024 4104 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:43.530895 4104 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.193:37097
I20260926 07:32:43.530946 4232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.193:37097 every 8 connection(s)
I20260926 07:32:43.531961 4104 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-0/data/info.pb
I20260926 07:32:43.532307 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 4104
I20260926 07:32:43.532455 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-0/wal/instance
I20260926 07:32:43.532821 22743 external_mini_cluster.cc:1428] Setting key 7a19da425006bf35bedba255d55f770c
WARNING: no policy specified for kudu/127.22.53.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.53.194@KRBTEST.COM" created.
Sep 26 07:32:43 dist-test-slave-wdjc krb5kdc[3997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407963, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.193@KRBTEST.COM for kudu/127.22.53.254@KRBTEST.COM
I20260926 07:32:43.552183 4081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:43.534264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.193:57565 (local address 127.22.53.254:35543)
0926 07:32:43.535987 (+ 1723us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:43.535995 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:43.536018 (+ 23us) server_negotiation.cc:407] Connection header received
0926 07:32:43.537733 (+ 1715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:43.537738 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:43.537802 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:43.537918 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:43.539350 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:43.539917 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:43.541318 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:43.541580 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:43.541895 (+ 315us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:43.547473 (+ 5578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:43.547480 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:43.547487 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:32:43.547540 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:43.549262 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:43.550132 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:43.550135 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:43.550137 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:43.550266 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:43.550783 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:43.550786 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:43.550787 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:43.550989 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:43.551072 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:43.551901 (+ 829us) server_negotiation.cc:299] Negotiation successful
0926 07:32:43.552038 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1599}
I20260926 07:32:43.553537 4235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:43.534776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:35543 (local address 127.22.53.193:57565)
0926 07:32:43.535670 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:43.535782 (+ 112us) client_negotiation.cc:174] Beginning negotiation
0926 07:32:43.537468 (+ 1686us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:32:43.538151 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:32:43.538159 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:32:43.538587 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:32:43.539161 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:43.539178 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:43.540089 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:43.540092 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:43.541163 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:43.541175 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:43.541952 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:43.541956 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:43.542168 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:43.542905 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:32:43.542935 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:32:43.547186 (+ 4251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:32:43.549437 (+ 2251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:32:43.549449 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:32:43.549474 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:32:43.549967 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:32:43.550396 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:32:43.550403 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:32:43.550407 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:32:43.550642 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:32:43.551109 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:32:43.551120 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:32:43.551540 (+ 420us) client_negotiation.cc:769] Sending connection context
0926 07:32:43.551812 (+ 272us) client_negotiation.cc:240] Negotiation successful
0926 07:32:43.552272 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":185,"threads_started":1}
I20260926 07:32:43.555150 4233 heartbeater.cc:347] Connected to a master server at 127.22.53.254:35543
I20260926 07:32:43.555481 4233 heartbeater.cc:464] Registering TS with master...
I20260926 07:32:43.556141 4233 heartbeater.cc:511] Master 127.22.53.254:35543 requested a full tablet report, sending...
I20260926 07:32:43.558151 4042 ts_manager.cc:194] Registered new tserver with Master: 84670aa208b740d1bdde3276c1a33264 (127.22.53.193:37097)
I20260926 07:32:43.559599 4042 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.53.193@KRBTEST.COM'} at 127.22.53.193:57565
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.53.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.194.keytab.
Entry for principal kudu/127.22.53.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.194.keytab.
I20260926 07:32:43.564755 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.53.194:0
--local_ip_for_outbound_sockets=127.22.53.194
--webserver_interface=127.22.53.194
--webserver_port=0
--tserver_master_addrs=127.22.53.254:35543
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.194.keytab
--principal=kudu/127.22.53.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:46315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:32:43.687278 4238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:32:43.687593 4238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:32:43.687640 4238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:32:43.692008 4238 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:32:43.692116 4238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:32:43.692225 4238 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.53.194
I20260926 07:32:43.698199 4238 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:46315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.194.keytab
--principal=kudu/127.22.53.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.53.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.53.194
--webserver_port=0
--tserver_master_addrs=127.22.53.254:35543
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4238
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.53.194
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:32:43.699457 4238 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:32:43.700664 4238 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:32:43.711532 4243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:32:43.711732 4238 server_base.cc:1034] running on GCE node
W20260926 07:32:43.711546 4246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:43.711975 4244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:32:43.712581 4238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:32:43.713373 4238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:32:43.714581 4238 hybrid_clock.cc:648] HybridClock initialized: now 1790407963714547 us; error 56 us; skew 500 ppm
Sep 26 07:32:43 dist-test-slave-wdjc krb5kdc[3997](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407963, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:32:43.718137 4238 init.cc:377] Logged in from keytab as kudu/127.22.53.194@KRBTEST.COM (short username kudu)
I20260926 07:32:43.719360 4238 webserver.cc:460] Webserver started at http://127.22.53.194:43399/ using document root <none> and password file <none>
I20260926 07:32:43.720078 4238 fs_manager.cc:359] Metadata directory not provided
I20260926 07:32:43.720180 4238 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:32:43.720429 4238 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:32:43.722728 4238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-1/data/instance:
uuid: "bc6ec79640c745879d85774d57594ab4"
format_stamp: "Formatted at 2026-09-26 07:32:43 on dist-test-slave-wdjc"
server_key: "ca7f2bc93ac94e8a8f22a2e9860eb593"
server_key_iv: "b22cde5e5514a23f3e64d49aef11e2fe"
server_key_version: "encryptionkey@0"
I20260926 07:32:43.723362 4238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-1/wal/instance:
uuid: "bc6ec79640c745879d85774d57594ab4"
format_stamp: "Formatted at 2026-09-26 07:32:43 on dist-test-slave-wdjc"
server_key: "ca7f2bc93ac94e8a8f22a2e9860eb593"
server_key_iv: "b22cde5e5514a23f3e64d49aef11e2fe"
server_key_version: "encryptionkey@0"
I20260926 07:32:43.727728 4238 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260926 07:32:43.730710 4253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:43.732200 4238 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 07:32:43.732354 4238 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-1/wal
uuid: "bc6ec79640c745879d85774d57594ab4"
format_stamp: "Formatted at 2026-09-26 07:32:43 on dist-test-slave-wdjc"
server_key: "ca7f2bc93ac94e8a8f22a2e9860eb593"
server_key_iv: "b22cde5e5514a23f3e64d49aef11e2fe"
server_key_version: "encryptionkey@0"
I20260926 07:32:43.732486 4238 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:32:43.760047 4238 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:32:43.763540 4238 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:32:43.763711 4238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:32:43.764401 4238 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:32:43.765410 4238 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:32:43.765460 4238 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:43.765532 4238 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:32:43.765569 4238 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:43.776026 4238 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.194:43777
I20260926 07:32:43.776043 4366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.194:43777 every 8 connection(s)
I20260926 07:32:43.777125 4238 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-1/data/info.pb
I20260926 07:32:43.783154 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 4238
I20260926 07:32:43.783305 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-1/wal/instance
I20260926 07:32:43.783667 22743 external_mini_cluster.cc:1428] Setting key e05501e310e364a0a50888c3ac249fb9
Sep 26 07:32:43 dist-test-slave-wdjc krb5kdc[3997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407963, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.194@KRBTEST.COM for kudu/127.22.53.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.22.53.195@KRBTEST.COM; defaulting to no policy
I20260926 07:32:43.794350 4081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:43.779189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.194:41591 (local address 127.22.53.254:35543)
0926 07:32:43.779355 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:43.779359 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:43.780307 (+ 948us) server_negotiation.cc:407] Connection header received
0926 07:32:43.781842 (+ 1535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:43.781846 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:43.781907 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:43.782039 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:43.783648 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:43.784325 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:43.786328 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:43.786578 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:43.786679 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:43.790226 (+ 3547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:43.790229 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:43.790232 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:32:43.790268 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:43.791824 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:43.792505 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:43.792508 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:43.792509 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:43.792653 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:43.793055 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:43.793059 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:43.793061 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:43.793283 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:43.793395 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:43.794008 (+ 613us) server_negotiation.cc:299] Negotiation successful
0926 07:32:43.794155 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 07:32:43.795433 4369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:43.779582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:35543 (local address 127.22.53.194:41591)
0926 07:32:43.780125 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:43.780172 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 07:32:43.781586 (+ 1414us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:32:43.782239 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:32:43.782247 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:32:43.782676 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:32:43.783411 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:43.783435 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:43.784483 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:43.784486 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:43.786090 (+ 1604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:43.786122 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:43.786772 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:43.786780 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:43.787050 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:43.788000 (+ 950us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:32:43.788019 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:32:43.789975 (+ 1956us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:32:43.791983 (+ 2008us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:32:43.791991 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:32:43.792011 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:32:43.792376 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:32:43.792764 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:32:43.792767 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:32:43.792769 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:32:43.792942 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:32:43.793397 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:32:43.793408 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:32:43.793732 (+ 324us) client_negotiation.cc:769] Sending connection context
0926 07:32:43.793946 (+ 214us) client_negotiation.cc:240] Negotiation successful
0926 07:32:43.794333 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":127,"threads_started":1}
I20260926 07:32:43.797092 4367 heartbeater.cc:347] Connected to a master server at 127.22.53.254:35543
I20260926 07:32:43.797356 4367 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.53.195@KRBTEST.COM" created.
I20260926 07:32:43.798028 4367 heartbeater.cc:511] Master 127.22.53.254:35543 requested a full tablet report, sending...
I20260926 07:32:43.799391 4042 ts_manager.cc:194] Registered new tserver with Master: bc6ec79640c745879d85774d57594ab4 (127.22.53.194:43777)
I20260926 07:32:43.800072 4042 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.53.194@KRBTEST.COM'} at 127.22.53.194:41591
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.53.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.195.keytab.
Entry for principal kudu/127.22.53.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.195.keytab.
I20260926 07:32:43.817029 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.53.195:0
--local_ip_for_outbound_sockets=127.22.53.195
--webserver_interface=127.22.53.195
--webserver_port=0
--tserver_master_addrs=127.22.53.254:35543
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.195.keytab
--principal=kudu/127.22.53.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:46315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:32:43.944336 4372 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:32:43.944679 4372 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:32:43.944769 4372 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:32:43.949542 4372 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:32:43.949632 4372 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:32:43.949699 4372 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.53.195
I20260926 07:32:43.955837 4372 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:46315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.195.keytab
--principal=kudu/127.22.53.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.53.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.53.195
--webserver_port=0
--tserver_master_addrs=127.22.53.254:35543
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4372
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.53.195
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:32:43.957192 4372 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:32:43.958349 4372 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:32:43.970117 4380 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:43.970131 4378 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:43.970126 4377 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:32:43.970381 4372 server_base.cc:1034] running on GCE node
I20260926 07:32:43.971032 4372 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:32:43.971832 4372 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:32:43.973057 4372 hybrid_clock.cc:648] HybridClock initialized: now 1790407963973008 us; error 63 us; skew 500 ppm
Sep 26 07:32:43 dist-test-slave-wdjc krb5kdc[3997](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407963, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:32:43.976279 4372 init.cc:377] Logged in from keytab as kudu/127.22.53.195@KRBTEST.COM (short username kudu)
I20260926 07:32:43.977306 4372 webserver.cc:460] Webserver started at http://127.22.53.195:41915/ using document root <none> and password file <none>
I20260926 07:32:43.977885 4372 fs_manager.cc:359] Metadata directory not provided
I20260926 07:32:43.977949 4372 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:32:43.978139 4372 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:32:43.980486 4372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-2/data/instance:
uuid: "2f290d8648c1444dba54d974cfa1a769"
format_stamp: "Formatted at 2026-09-26 07:32:43 on dist-test-slave-wdjc"
server_key: "481e228797e50ef9032d222ff42d3a8b"
server_key_iv: "3405b1c091992969b8ffbec0bbba738b"
server_key_version: "encryptionkey@0"
I20260926 07:32:43.981148 4372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-2/wal/instance:
uuid: "2f290d8648c1444dba54d974cfa1a769"
format_stamp: "Formatted at 2026-09-26 07:32:43 on dist-test-slave-wdjc"
server_key: "481e228797e50ef9032d222ff42d3a8b"
server_key_iv: "3405b1c091992969b8ffbec0bbba738b"
server_key_version: "encryptionkey@0"
I20260926 07:32:43.985558 4372 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 07:32:43.988634 4387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:43.990211 4372 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 07:32:43.990340 4372 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-2/wal
uuid: "2f290d8648c1444dba54d974cfa1a769"
format_stamp: "Formatted at 2026-09-26 07:32:43 on dist-test-slave-wdjc"
server_key: "481e228797e50ef9032d222ff42d3a8b"
server_key_iv: "3405b1c091992969b8ffbec0bbba738b"
server_key_version: "encryptionkey@0"
I20260926 07:32:43.990455 4372 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:32:44.022542 4372 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:32:44.026932 4372 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:32:44.027160 4372 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:32:44.027832 4372 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:32:44.028874 4372 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:32:44.028939 4372 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:44.028985 4372 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:32:44.029001 4372 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:44.039345 4372 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.195:38919
I20260926 07:32:44.039361 4500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.195:38919 every 8 connection(s)
I20260926 07:32:44.040405 4372 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-2/data/info.pb
I20260926 07:32:44.045995 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 4372
I20260926 07:32:44.046154 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-2/wal/instance
I20260926 07:32:44.046449 22743 external_mini_cluster.cc:1428] Setting key 623408adbdcf24d329070805de0710a1
Sep 26 07:32:44 dist-test-slave-wdjc krb5kdc[3997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407963, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.195@KRBTEST.COM for kudu/127.22.53.254@KRBTEST.COM
I20260926 07:32:44.057307 4081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:44.042610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.195:57719 (local address 127.22.53.254:35543)
0926 07:32:44.042825 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:44.042829 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:44.043751 (+ 922us) server_negotiation.cc:407] Connection header received
0926 07:32:44.045428 (+ 1677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:44.045432 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:44.045503 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:44.045627 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:44.047228 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:44.047789 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:44.049306 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:44.049544 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:44.049671 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:44.052814 (+ 3143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:44.052818 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:44.052820 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:32:44.052850 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:44.054674 (+ 1824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:44.055378 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:44.055384 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:44.055391 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:44.055482 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:44.055910 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:44.055914 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:44.055916 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:44.056148 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:44.056269 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:44.056932 (+ 663us) server_negotiation.cc:299] Negotiation successful
0926 07:32:44.057095 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260926 07:32:44.058310 4503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:44.042994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:35543 (local address 127.22.53.195:57719)
0926 07:32:44.043538 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:44.043612 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0926 07:32:44.045148 (+ 1536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:32:44.045863 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:32:44.045870 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:32:44.046375 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:32:44.047050 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:44.047069 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:44.047955 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:44.047958 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:44.049153 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:44.049168 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:44.049726 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:44.049730 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:44.049931 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:44.050584 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:32:44.050614 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:32:44.052470 (+ 1856us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:32:44.054844 (+ 2374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:32:44.054852 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:32:44.054879 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:32:44.055226 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:32:44.055606 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:32:44.055609 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:32:44.055611 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:32:44.055787 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:32:44.056269 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:32:44.056278 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:32:44.056658 (+ 380us) client_negotiation.cc:769] Sending connection context
0926 07:32:44.056870 (+ 212us) client_negotiation.cc:240] Negotiation successful
0926 07:32:44.057311 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":147,"threads_started":1}
I20260926 07:32:44.060990 4501 heartbeater.cc:347] Connected to a master server at 127.22.53.254:35543
I20260926 07:32:44.061295 4501 heartbeater.cc:464] Registering TS with master...
I20260926 07:32:44.061967 4501 heartbeater.cc:511] Master 127.22.53.254:35543 requested a full tablet report, sending...
I20260926 07:32:44.063374 4039 ts_manager.cc:194] Registered new tserver with Master: 2f290d8648c1444dba54d974cfa1a769 (127.22.53.195:38919)
I20260926 07:32:44.064414 4039 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.53.195@KRBTEST.COM'} at 127.22.53.195:57719
I20260926 07:32:44.074647 22743 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:32:44.086761 4081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:44.077022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40154 (local address 127.22.53.254:35543)
0926 07:32:44.077301 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:44.077307 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:44.077472 (+ 165us) server_negotiation.cc:407] Connection header received
0926 07:32:44.077658 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:44.077664 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:44.077717 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:44.077820 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:44.078612 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:44.079185 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:44.080155 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:44.080394 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:44.080517 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:44.082659 (+ 2142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:44.082663 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:44.082666 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:32:44.082704 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:44.084783 (+ 2079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:44.085370 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:44.085373 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:44.085374 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:44.085479 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:44.085822 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:44.085824 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:44.085826 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:44.086078 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:44.086194 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:44.086488 (+ 294us) server_negotiation.cc:299] Negotiation successful
0926 07:32:44.086618 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163}
I20260926 07:32:44.090379 4039 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40154:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:32:44.093948 4039 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:32:44.110981 4513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:44.104516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.195:38919 (local address 127.0.0.1:57952)
0926 07:32:44.105540 (+ 1024us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:44.105557 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 07:32:44.105648 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:32:44.106315 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:32:44.106318 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:32:44.106341 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:32:44.106678 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:44.106685 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:44.108217 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:44.108220 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:44.109386 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:44.109396 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:44.110349 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:44.110353 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:44.110462 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:44.110501 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 07:32:44.110636 (+ 135us) client_negotiation.cc:240] Negotiation successful
0926 07:32:44.110789 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":961,"mutex_wait_us":271,"spinlock_wait_cycles":2944,"thread_start_us":102,"threads_started":1}
I20260926 07:32:44.111006 4512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:44.104284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.194:43777 (local address 127.0.0.1:48366)
0926 07:32:44.104950 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:44.105009 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 07:32:44.105185 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:32:44.106244 (+ 1059us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:32:44.106253 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:32:44.106302 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:32:44.106492 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:44.106502 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:44.108175 (+ 1673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:44.108178 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:44.109442 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:44.109448 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:44.110652 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:44.110656 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:44.110743 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:44.110765 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 07:32:44.110843 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 07:32:44.110904 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"mutex_wait_us":59,"spinlock_wait_cycles":6144,"thread_start_us":118,"threads_started":1}
I20260926 07:32:44.111419 4517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:44.105252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57952 (local address 127.22.53.195:38919)
0926 07:32:44.105906 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:44.105916 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:44.105936 (+ 20us) server_negotiation.cc:407] Connection header received
0926 07:32:44.105999 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:44.106003 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:44.106180 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:44.106463 (+ 283us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:32:44.106812 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:44.108024 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:44.109533 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:44.110196 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:44.110334 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:44.110505 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:44.110869 (+ 364us) server_negotiation.cc:299] Negotiation successful
0926 07:32:44.110996 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":150,"threads_started":1}
I20260926 07:32:44.111603 4516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:44.105259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48366 (local address 127.22.53.194:43777)
0926 07:32:44.105655 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:44.105671 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:44.105701 (+ 30us) server_negotiation.cc:407] Connection header received
0926 07:32:44.105793 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:44.105801 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:44.106044 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:44.106668 (+ 624us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:32:44.106842 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:44.108024 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:44.109602 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:44.110454 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:44.110616 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:44.110762 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:44.110893 (+ 131us) server_negotiation.cc:299] Negotiation successful
0926 07:32:44.110987 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":145,"threads_started":1}
I20260926 07:32:44.111804 4515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:44.104288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.193:37097 (local address 127.0.0.1:39536)
0926 07:32:44.105128 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:44.105140 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 07:32:44.105211 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:32:44.106252 (+ 1041us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:32:44.106255 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:32:44.106294 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:32:44.106715 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:44.106722 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:44.107886 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:44.107892 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:44.109437 (+ 1545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:44.109448 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:44.111361 (+ 1913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:44.111367 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:44.111436 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:44.111454 (+ 18us) client_negotiation.cc:769] Sending connection context
0926 07:32:44.111556 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 07:32:44.111641 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":783,"mutex_wait_us":288,"spinlock_wait_cycles":1280,"thread_start_us":125,"threads_started":1}
I20260926 07:32:44.112448 4514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:44.104409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39536 (local address 127.22.53.193:37097)
0926 07:32:44.105770 (+ 1361us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:44.105777 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:44.105804 (+ 27us) server_negotiation.cc:407] Connection header received
0926 07:32:44.105891 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:44.105897 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:44.106094 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:44.106464 (+ 370us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:32:44.106849 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:44.107758 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:44.109602 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:44.110455 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:44.110616 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:44.110762 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:44.112158 (+ 1396us) server_negotiation.cc:299] Negotiation successful
0926 07:32:44.112285 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1225,"thread_start_us":133,"threads_started":1}
I20260926 07:32:44.114886 4435 tablet_service.cc:1467] Processing CreateTablet for tablet 5565e3d3c5ed44f29c9755c4d598d3c8 (DEFAULT_TABLE table=test-table [id=74398147aff446a0bf197dbcb3f49c79]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:32:44.114888 4301 tablet_service.cc:1467] Processing CreateTablet for tablet 5565e3d3c5ed44f29c9755c4d598d3c8 (DEFAULT_TABLE table=test-table [id=74398147aff446a0bf197dbcb3f49c79]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:32:44.116269 4301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5565e3d3c5ed44f29c9755c4d598d3c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:32:44.116269 4435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5565e3d3c5ed44f29c9755c4d598d3c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:32:44.115613 4167 tablet_service.cc:1467] Processing CreateTablet for tablet 5565e3d3c5ed44f29c9755c4d598d3c8 (DEFAULT_TABLE table=test-table [id=74398147aff446a0bf197dbcb3f49c79]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:32:44.117295 4167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5565e3d3c5ed44f29c9755c4d598d3c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:32:44.125106 4520 tablet_bootstrap.cc:492] T 5565e3d3c5ed44f29c9755c4d598d3c8 P 84670aa208b740d1bdde3276c1a33264: Bootstrap starting.
I20260926 07:32:44.125639 4519 tablet_bootstrap.cc:492] T 5565e3d3c5ed44f29c9755c4d598d3c8 P 2f290d8648c1444dba54d974cfa1a769: Bootstrap starting.
I20260926 07:32:44.126108 4518 tablet_bootstrap.cc:492] T 5565e3d3c5ed44f29c9755c4d598d3c8 P bc6ec79640c745879d85774d57594ab4: Bootstrap starting.
I20260926 07:32:44.128746 4520 tablet_bootstrap.cc:654] T 5565e3d3c5ed44f29c9755c4d598d3c8 P 84670aa208b740d1bdde3276c1a33264: Neither blocks nor log segments found. Creating new log.
I20260926 07:32:44.128780 4519 tablet_bootstrap.cc:654] T 5565e3d3c5ed44f29c9755c4d598d3c8 P 2f290d8648c1444dba54d974cfa1a769: Neither blocks nor log segments found. Creating new log.
I20260926 07:32:44.129383 4518 tablet_bootstrap.cc:654] T 5565e3d3c5ed44f29c9755c4d598d3c8 P bc6ec79640c745879d85774d57594ab4: Neither blocks nor log segments found. Creating new log.
I20260926 07:32:44.129776 4520 log.cc:824] T 5565e3d3c5ed44f29c9755c4d598d3c8 P 84670aa208b740d1bdde3276c1a33264: Log is configured to *not* fsync() on all Append() calls
I20260926 07:32:44.130188 4518 log.cc:824] T 5565e3d3c5ed44f29c9755c4d598d3c8 P bc6ec79640c745879d85774d57594ab4: Log is configured to *not* fsync() on all Append() calls
I20260926 07:32:44.130188 4519 log.cc:824] T 5565e3d3c5ed44f29c9755c4d598d3c8 P 2f290d8648c1444dba54d974cfa1a769: Log is configured to *not* fsync() on all Append() calls
I20260926 07:32:44.132580 4519 tablet_bootstrap.cc:492] T 5565e3d3c5ed44f29c9755c4d598d3c8 P 2f290d8648c1444dba54d974cfa1a769: No bootstrap required, opened a new log
I20260926 07:32:44.132581 4518 tablet_bootstrap.cc:492] T 5565e3d3c5ed44f29c9755c4d598d3c8 P bc6ec79640c745879d85774d57594ab4: No bootstrap required, opened a new log
I20260926 07:32:44.132609 4520 tablet_bootstrap.cc:492] T 5565e3d3c5ed44f29c9755c4d598d3c8 P 84670aa208b740d1bdde3276c1a33264: No bootstrap required, opened a new log
I20260926 07:32:44.132827 4518 ts_tablet_manager.cc:1397] T 5565e3d3c5ed44f29c9755c4d598d3c8 P bc6ec79640c745879d85774d57594ab4: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260926 07:32:44.132827 4519 ts_tablet_manager.cc:1397] T 5565e3d3c5ed44f29c9755c4d598d3c8 P 2f290d8648c1444dba54d974cfa1a769: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260926 07:32:44.132939 4520 ts_tablet_manager.cc:1397] T 5565e3d3c5ed44f29c9755c4d598d3c8 P 84670aa208b740d1bdde3276c1a33264: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.004s
I20260926 07:32:44.137373 4518 raft_consensus.cc:348] T 5565e3d3c5ed44f29c9755c4d598d3c8 P bc6ec79640c745879d85774d57594ab4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84670aa208b740d1bdde3276c1a33264" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 37097 } } peers { permanent_uuid: "bc6ec79640c745879d85774d57594ab4" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 43777 } } peers { permanent_uuid: "2f290d8648c1444dba54d974cfa1a769" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 38919 } }
I20260926 07:32:44.137374 4519 raft_consensus.cc:348] T 5565e3d3c5ed44f29c9755c4d598d3c8 P 2f290d8648c1444dba54d974cfa1a769 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84670aa208b740d1bdde3276c1a33264" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 37097 } } peers { permanent_uuid: "bc6ec79640c745879d85774d57594ab4" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 43777 } } peers { permanent_uuid: "2f290d8648c1444dba54d974cfa1a769" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 38919 } }
I20260926 07:32:44.137351 4520 raft_consensus.cc:348] T 5565e3d3c5ed44f29c9755c4d598d3c8 P 84670aa208b740d1bdde3276c1a33264 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84670aa208b740d1bdde3276c1a33264" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 37097 } } peers { permanent_uuid: "bc6ec79640c745879d85774d57594ab4" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 43777 } } peers { permanent_uuid: "2f290d8648c1444dba54d974cfa1a769" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 38919 } }
I20260926 07:32:44.137698 4518 raft_consensus.cc:374] T 5565e3d3c5ed44f29c9755c4d598d3c8 P bc6ec79640c745879d85774d57594ab4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:32:44.137698 4519 raft_consensus.cc:374] T 5565e3d3c5ed44f29c9755c4d598d3c8 P 2f290d8648c1444dba54d974cfa1a769 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:32:44.137748 4520 raft_consensus.cc:374] T 5565e3d3c5ed44f29c9755c4d598d3c8 P 84670aa208b740d1bdde3276c1a33264 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:32:44.137764 4518 raft_consensus.cc:729] T 5565e3d3c5ed44f29c9755c4d598d3c8 P bc6ec79640c745879d85774d57594ab4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc6ec79640c745879d85774d57594ab4, State: Initialized, Role: FOLLOWER
I20260926 07:32:44.137764 4519 raft_consensus.cc:729] T 5565e3d3c5ed44f29c9755c4d598d3c8 P 2f290d8648c1444dba54d974cfa1a769 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f290d8648c1444dba54d974cfa1a769, State: Initialized, Role: FOLLOWER
I20260926 07:32:44.137812 4520 raft_consensus.cc:729] T 5565e3d3c5ed44f29c9755c4d598d3c8 P 84670aa208b740d1bdde3276c1a33264 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84670aa208b740d1bdde3276c1a33264, State: Initialized, Role: FOLLOWER
I20260926 07:32:44.138294 4520 consensus_queue.cc:260] T 5565e3d3c5ed44f29c9755c4d598d3c8 P 84670aa208b740d1bdde3276c1a33264 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84670aa208b740d1bdde3276c1a33264" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 37097 } } peers { permanent_uuid: "bc6ec79640c745879d85774d57594ab4" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 43777 } } peers { permanent_uuid: "2f290d8648c1444dba54d974cfa1a769" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 38919 } }
I20260926 07:32:44.138298 4519 consensus_queue.cc:260] T 5565e3d3c5ed44f29c9755c4d598d3c8 P 2f290d8648c1444dba54d974cfa1a769 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84670aa208b740d1bdde3276c1a33264" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 37097 } } peers { permanent_uuid: "bc6ec79640c745879d85774d57594ab4" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 43777 } } peers { permanent_uuid: "2f290d8648c1444dba54d974cfa1a769" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 38919 } }
I20260926 07:32:44.138298 4518 consensus_queue.cc:260] T 5565e3d3c5ed44f29c9755c4d598d3c8 P bc6ec79640c745879d85774d57594ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84670aa208b740d1bdde3276c1a33264" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 37097 } } peers { permanent_uuid: "bc6ec79640c745879d85774d57594ab4" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 43777 } } peers { permanent_uuid: "2f290d8648c1444dba54d974cfa1a769" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 38919 } }
I20260926 07:32:44.139364 4367 heartbeater.cc:503] Master 127.22.53.254:35543 was elected leader, sending a full tablet report...
I20260926 07:32:44.139536 4518 ts_tablet_manager.cc:1428] T 5565e3d3c5ed44f29c9755c4d598d3c8 P bc6ec79640c745879d85774d57594ab4: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260926 07:32:44.139542 4520 ts_tablet_manager.cc:1428] T 5565e3d3c5ed44f29c9755c4d598d3c8 P 84670aa208b740d1bdde3276c1a33264: Time spent starting tablet: real 0.007s user 0.004s sys 0.002s
I20260926 07:32:44.139613 4233 heartbeater.cc:503] Master 127.22.53.254:35543 was elected leader, sending a full tablet report...
I20260926 07:32:44.139961 4501 heartbeater.cc:503] Master 127.22.53.254:35543 was elected leader, sending a full tablet report...
I20260926 07:32:44.140208 4519 ts_tablet_manager.cc:1428] T 5565e3d3c5ed44f29c9755c4d598d3c8 P 2f290d8648c1444dba54d974cfa1a769: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260926 07:32:44.199713 4524 raft_consensus.cc:482] T 5565e3d3c5ed44f29c9755c4d598d3c8 P bc6ec79640c745879d85774d57594ab4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:32:44.199992 4524 raft_consensus.cc:504] T 5565e3d3c5ed44f29c9755c4d598d3c8 P bc6ec79640c745879d85774d57594ab4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84670aa208b740d1bdde3276c1a33264" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 37097 } } peers { permanent_uuid: "bc6ec79640c745879d85774d57594ab4" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 43777 } } peers { permanent_uuid: "2f290d8648c1444dba54d974cfa1a769" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 38919 } }
I20260926 07:32:44.201370 4524 leader_election.cc:288] T 5565e3d3c5ed44f29c9755c4d598d3c8 P bc6ec79640c745879d85774d57594ab4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84670aa208b740d1bdde3276c1a33264 (127.22.53.193:37097), 2f290d8648c1444dba54d974cfa1a769 (127.22.53.195:38919)
I20260926 07:32:44.207926 4517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:44.201802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.194:60381 (local address 127.22.53.195:38919)
0926 07:32:44.201957 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:44.201961 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:44.201979 (+ 18us) server_negotiation.cc:407] Connection header received
0926 07:32:44.202200 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:44.202204 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:44.202259 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:44.202596 (+ 337us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:32:44.202959 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:44.203837 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:44.206300 (+ 2463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:44.207005 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:44.207153 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:44.207199 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:44.207654 (+ 455us) server_negotiation.cc:299] Negotiation successful
0926 07:32:44.207769 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 07:32:44.207926 4514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:44.201892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.194:45431 (local address 127.22.53.193:37097)
0926 07:32:44.202049 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:44.202054 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:44.202602 (+ 548us) server_negotiation.cc:407] Connection header received
0926 07:32:44.202807 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:44.202812 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:44.202873 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:44.202930 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:32:44.203310 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:44.203857 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:44.206211 (+ 2354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:44.206953 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:44.207134 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:44.207196 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:44.207654 (+ 458us) server_negotiation.cc:299] Negotiation successful
0926 07:32:44.207769 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 07:32:44.207971 4369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:44.201641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.195:38919 (local address 127.22.53.194:60381)
0926 07:32:44.201891 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:44.201908 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 07:32:44.202036 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:32:44.202609 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:32:44.202612 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:32:44.202638 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:32:44.202802 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:44.202808 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:44.204077 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:44.204081 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:44.206006 (+ 1925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:44.206029 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:44.207207 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:44.207212 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:44.207407 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:44.207467 (+ 60us) client_negotiation.cc:769] Sending connection context
0926 07:32:44.207598 (+ 131us) client_negotiation.cc:240] Negotiation successful
0926 07:32:44.207750 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"mutex_wait_us":307}
I20260926 07:32:44.207985 4527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:44.201651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.193:37097 (local address 127.22.53.194:45431)
0926 07:32:44.202496 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:44.202507 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 07:32:44.202664 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:32:44.203069 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:32:44.203070 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:32:44.203086 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:32:44.203187 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:44.203191 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:44.203994 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:44.204000 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:44.206006 (+ 2006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:44.206029 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:44.207207 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:44.207212 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:44.207409 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:44.207471 (+ 62us) client_negotiation.cc:769] Sending connection context
0926 07:32:44.207602 (+ 131us) client_negotiation.cc:240] Negotiation successful
0926 07:32:44.207751 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":785,"mutex_wait_us":638,"thread_start_us":148,"threads_started":1}
I20260926 07:32:44.208910 4455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5565e3d3c5ed44f29c9755c4d598d3c8" candidate_uuid: "bc6ec79640c745879d85774d57594ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f290d8648c1444dba54d974cfa1a769" is_pre_election: true
I20260926 07:32:44.208997 4187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5565e3d3c5ed44f29c9755c4d598d3c8" candidate_uuid: "bc6ec79640c745879d85774d57594ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84670aa208b740d1bdde3276c1a33264" is_pre_election: true
I20260926 07:32:44.209313 4455 raft_consensus.cc:2454] T 5565e3d3c5ed44f29c9755c4d598d3c8 P 2f290d8648c1444dba54d974cfa1a769 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc6ec79640c745879d85774d57594ab4 in term 0.
I20260926 07:32:44.209360 4187 raft_consensus.cc:2454] T 5565e3d3c5ed44f29c9755c4d598d3c8 P 84670aa208b740d1bdde3276c1a33264 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc6ec79640c745879d85774d57594ab4 in term 0.
I20260926 07:32:44.209923 4254 leader_election.cc:302] T 5565e3d3c5ed44f29c9755c4d598d3c8 P bc6ec79640c745879d85774d57594ab4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f290d8648c1444dba54d974cfa1a769, bc6ec79640c745879d85774d57594ab4; no voters:
I20260926 07:32:44.210307 4524 raft_consensus.cc:2781] T 5565e3d3c5ed44f29c9755c4d598d3c8 P bc6ec79640c745879d85774d57594ab4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:32:44.210407 4524 raft_consensus.cc:482] T 5565e3d3c5ed44f29c9755c4d598d3c8 P bc6ec79640c745879d85774d57594ab4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:32:44.210453 4524 raft_consensus.cc:3037] T 5565e3d3c5ed44f29c9755c4d598d3c8 P bc6ec79640c745879d85774d57594ab4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:32:44.211665 4524 raft_consensus.cc:504] T 5565e3d3c5ed44f29c9755c4d598d3c8 P bc6ec79640c745879d85774d57594ab4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84670aa208b740d1bdde3276c1a33264" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 37097 } } peers { permanent_uuid: "bc6ec79640c745879d85774d57594ab4" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 43777 } } peers { permanent_uuid: "2f290d8648c1444dba54d974cfa1a769" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 38919 } }
I20260926 07:32:44.212092 4524 leader_election.cc:288] T 5565e3d3c5ed44f29c9755c4d598d3c8 P bc6ec79640c745879d85774d57594ab4 [CANDIDATE]: Term 1 election: Requested vote from peers 84670aa208b740d1bdde3276c1a33264 (127.22.53.193:37097), 2f290d8648c1444dba54d974cfa1a769 (127.22.53.195:38919)
I20260926 07:32:44.212687 4187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5565e3d3c5ed44f29c9755c4d598d3c8" candidate_uuid: "bc6ec79640c745879d85774d57594ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84670aa208b740d1bdde3276c1a33264"
I20260926 07:32:44.212687 4455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5565e3d3c5ed44f29c9755c4d598d3c8" candidate_uuid: "bc6ec79640c745879d85774d57594ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f290d8648c1444dba54d974cfa1a769"
I20260926 07:32:44.212822 4187 raft_consensus.cc:3037] T 5565e3d3c5ed44f29c9755c4d598d3c8 P 84670aa208b740d1bdde3276c1a33264 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:32:44.212854 4455 raft_consensus.cc:3037] T 5565e3d3c5ed44f29c9755c4d598d3c8 P 2f290d8648c1444dba54d974cfa1a769 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:32:44.214072 4455 raft_consensus.cc:2454] T 5565e3d3c5ed44f29c9755c4d598d3c8 P 2f290d8648c1444dba54d974cfa1a769 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc6ec79640c745879d85774d57594ab4 in term 1.
I20260926 07:32:44.214082 4187 raft_consensus.cc:2454] T 5565e3d3c5ed44f29c9755c4d598d3c8 P 84670aa208b740d1bdde3276c1a33264 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc6ec79640c745879d85774d57594ab4 in term 1.
I20260926 07:32:44.214510 4254 leader_election.cc:302] T 5565e3d3c5ed44f29c9755c4d598d3c8 P bc6ec79640c745879d85774d57594ab4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f290d8648c1444dba54d974cfa1a769, bc6ec79640c745879d85774d57594ab4; no voters:
I20260926 07:32:44.214726 4524 raft_consensus.cc:2781] T 5565e3d3c5ed44f29c9755c4d598d3c8 P bc6ec79640c745879d85774d57594ab4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:32:44.214972 4524 raft_consensus.cc:686] T 5565e3d3c5ed44f29c9755c4d598d3c8 P bc6ec79640c745879d85774d57594ab4 [term 1 LEADER]: Becoming Leader. State: Replica: bc6ec79640c745879d85774d57594ab4, State: Running, Role: LEADER
I20260926 07:32:44.215368 4524 consensus_queue.cc:237] T 5565e3d3c5ed44f29c9755c4d598d3c8 P bc6ec79640c745879d85774d57594ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84670aa208b740d1bdde3276c1a33264" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 37097 } } peers { permanent_uuid: "bc6ec79640c745879d85774d57594ab4" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 43777 } } peers { permanent_uuid: "2f290d8648c1444dba54d974cfa1a769" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 38919 } }
I20260926 07:32:44.219223 4042 catalog_manager.cc:5495] T 5565e3d3c5ed44f29c9755c4d598d3c8 P bc6ec79640c745879d85774d57594ab4 reported cstate change: term changed from 0 to 1, leader changed from <none> to bc6ec79640c745879d85774d57594ab4 (127.22.53.194). New cstate: current_term: 1 leader_uuid: "bc6ec79640c745879d85774d57594ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84670aa208b740d1bdde3276c1a33264" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 37097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc6ec79640c745879d85774d57594ab4" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 43777 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f290d8648c1444dba54d974cfa1a769" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 38919 } health_report { overall_health: UNKNOWN } } }
I20260926 07:32:44.240646 4516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:44.236995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48382 (local address 127.22.53.194:43777)
0926 07:32:44.237203 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:44.237207 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:44.237227 (+ 20us) server_negotiation.cc:407] Connection header received
0926 07:32:44.237345 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:44.237347 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:44.237407 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:44.237505 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:32:44.237771 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:44.238276 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:44.239140 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:44.239351 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:44.239430 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:44.239726 (+ 296us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:32:44.240188 (+ 462us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:32:44.240289 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:44.240349 (+ 60us) server_negotiation.cc:299] Negotiation successful
0926 07:32:44.240410 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260926 07:32:44.247004 4455 raft_consensus.cc:1261] T 5565e3d3c5ed44f29c9755c4d598d3c8 P 2f290d8648c1444dba54d974cfa1a769 [term 1 FOLLOWER]: Refusing update from remote peer bc6ec79640c745879d85774d57594ab4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:32:44.247005 4187 raft_consensus.cc:1261] T 5565e3d3c5ed44f29c9755c4d598d3c8 P 84670aa208b740d1bdde3276c1a33264 [term 1 FOLLOWER]: Refusing update from remote peer bc6ec79640c745879d85774d57594ab4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:32:44.247780 4528 consensus_queue.cc:1035] T 5565e3d3c5ed44f29c9755c4d598d3c8 P bc6ec79640c745879d85774d57594ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84670aa208b740d1bdde3276c1a33264" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 37097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:32:44.248019 4524 consensus_queue.cc:1035] T 5565e3d3c5ed44f29c9755c4d598d3c8 P bc6ec79640c745879d85774d57594ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f290d8648c1444dba54d974cfa1a769" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 38919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:32:44.255446 4531 mvcc.cc:205] Tried to move back new op lower bound from 7333511021547794432 to 7333511021429215232. Current Snapshot: MvccSnapshot[applied={T|T < 7333511021547794432}]
I20260926 07:32:44.255702 4533 mvcc.cc:205] Tried to move back new op lower bound from 7333511021547794432 to 7333511021429215232. Current Snapshot: MvccSnapshot[applied={T|T < 7333511021547794432}]
I20260926 07:32:44.257570 4532 mvcc.cc:205] Tried to move back new op lower bound from 7333511021547794432 to 7333511021429215232. Current Snapshot: MvccSnapshot[applied={T|T < 7333511021547794432}]
I20260926 07:32:44.266041 4041 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40154:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:32:44.266229 4041 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40154:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:32:44.269343 4041 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 095dfa56bdf344f6b50d5cc18be740cd: Sending DeleteTablet for 3 replicas of tablet 5565e3d3c5ed44f29c9755c4d598d3c8
I20260926 07:32:44.270351 4167 tablet_service.cc:1514] Processing DeleteTablet for tablet 5565e3d3c5ed44f29c9755c4d598d3c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:32:44 UTC) from {username='kudu', principal='kudu/127.22.53.254@KRBTEST.COM'} at 127.0.0.1:39536
I20260926 07:32:44.270430 4435 tablet_service.cc:1514] Processing DeleteTablet for tablet 5565e3d3c5ed44f29c9755c4d598d3c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:32:44 UTC) from {username='kudu', principal='kudu/127.22.53.254@KRBTEST.COM'} at 127.0.0.1:57952
I20260926 07:32:44.270812 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 4104
I20260926 07:32:44.270964 4301 tablet_service.cc:1514] Processing DeleteTablet for tablet 5565e3d3c5ed44f29c9755c4d598d3c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:32:44 UTC) from {username='kudu', principal='kudu/127.22.53.254@KRBTEST.COM'} at 127.0.0.1:48366
I20260926 07:32:44.271673 4542 tablet_replica.cc:330] T 5565e3d3c5ed44f29c9755c4d598d3c8 P 2f290d8648c1444dba54d974cfa1a769: stopping tablet replica
I20260926 07:32:44.272074 4542 raft_consensus.cc:2229] T 5565e3d3c5ed44f29c9755c4d598d3c8 P 2f290d8648c1444dba54d974cfa1a769 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:32:44.272432 4542 raft_consensus.cc:2258] T 5565e3d3c5ed44f29c9755c4d598d3c8 P 2f290d8648c1444dba54d974cfa1a769 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:32:44.272667 4543 tablet_replica.cc:330] T 5565e3d3c5ed44f29c9755c4d598d3c8 P bc6ec79640c745879d85774d57594ab4: stopping tablet replica
I20260926 07:32:44.272989 4543 raft_consensus.cc:2229] T 5565e3d3c5ed44f29c9755c4d598d3c8 P bc6ec79640c745879d85774d57594ab4 [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:32:44.273416 4543 raft_consensus.cc:2258] T 5565e3d3c5ed44f29c9755c4d598d3c8 P bc6ec79640c745879d85774d57594ab4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:32:44.274307 4542 ts_tablet_manager.cc:1905] T 5565e3d3c5ed44f29c9755c4d598d3c8 P 2f290d8648c1444dba54d974cfa1a769: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:32:44.275254 4543 ts_tablet_manager.cc:1905] T 5565e3d3c5ed44f29c9755c4d598d3c8 P bc6ec79640c745879d85774d57594ab4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:32:44.278231 4543 ts_tablet_manager.cc:1918] T 5565e3d3c5ed44f29c9755c4d598d3c8 P bc6ec79640c745879d85774d57594ab4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:32:44.278369 4543 log.cc:1196] T 5565e3d3c5ed44f29c9755c4d598d3c8 P bc6ec79640c745879d85774d57594ab4: Deleting WAL directory at /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-1/wal/wals/5565e3d3c5ed44f29c9755c4d598d3c8
I20260926 07:32:44.278713 4543 ts_tablet_manager.cc:1939] T 5565e3d3c5ed44f29c9755c4d598d3c8 P bc6ec79640c745879d85774d57594ab4: Deleting consensus metadata
I20260926 07:32:44.278884 4542 ts_tablet_manager.cc:1918] T 5565e3d3c5ed44f29c9755c4d598d3c8 P 2f290d8648c1444dba54d974cfa1a769: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:32:44.279156 4542 log.cc:1196] T 5565e3d3c5ed44f29c9755c4d598d3c8 P 2f290d8648c1444dba54d974cfa1a769: Deleting WAL directory at /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-2/wal/wals/5565e3d3c5ed44f29c9755c4d598d3c8
I20260926 07:32:44.279670 4542 ts_tablet_manager.cc:1939] T 5565e3d3c5ed44f29c9755c4d598d3c8 P 2f290d8648c1444dba54d974cfa1a769: Deleting consensus metadata
W20260926 07:32:44.280027 4027 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.53.193:37097 (error 108)
I20260926 07:32:44.280061 4028 catalog_manager.cc:4841] TS bc6ec79640c745879d85774d57594ab4 (127.22.53.194:43777): tablet 5565e3d3c5ed44f29c9755c4d598d3c8 (table test-table [id=74398147aff446a0bf197dbcb3f49c79]) successfully deleted
I20260926 07:32:44.280333 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 4238
I20260926 07:32:44.283032 4026 catalog_manager.cc:4841] TS 2f290d8648c1444dba54d974cfa1a769 (127.22.53.195:38919): tablet 5565e3d3c5ed44f29c9755c4d598d3c8 (table test-table [id=74398147aff446a0bf197dbcb3f49c79]) successfully deleted
I20260926 07:32:44.283831 4515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:44.283470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.193:37097 (local address 127.0.0.1:39552)
0926 07:32:44.283689 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:44.283754 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.193:37097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131}
W20260926 07:32:44.284071 4027 catalog_manager.cc:4584] TS 84670aa208b740d1bdde3276c1a33264 (127.22.53.193:37097): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5565e3d3c5ed44f29c9755c4d598d3c8: Network error: Client connection negotiation failed: client connection to 127.22.53.193:37097: connect: Connection refused (error 111)
I20260926 07:32:44.286393 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 4372
I20260926 07:32:44.292727 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 4010
I20260926 07:32:44.299616 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.53.254:35543
--webserver_interface=127.22.53.254
--webserver_port=37215
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab
--principal=kudu/127.22.53.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:46315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.53.254:35543
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:32:44.418864 4545 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:32:44.419190 4545 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:32:44.419234 4545 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:32:44.423656 4545 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:32:44.423750 4545 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:32:44.423770 4545 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:32:44.423787 4545 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:32:44.423804 4545 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:32:44.429678 4545 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:46315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.22.53.254:35543
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab
--principal=kudu/127.22.53.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.53.254:35543
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.53.254
--webserver_port=37215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4545
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:32:44.430922 4545 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:32:44.432036 4545 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:32:44.446184 4552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:44.446192 4554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:44.446184 4551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:32:44.446600 4545 server_base.cc:1034] running on GCE node
I20260926 07:32:44.447297 4545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:32:44.448711 4545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:32:44.449913 4545 hybrid_clock.cc:648] HybridClock initialized: now 1790407964449900 us; error 36 us; skew 500 ppm
Sep 26 07:32:44 dist-test-slave-wdjc krb5kdc[3997](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407964, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:32:44.452936 4545 init.cc:377] Logged in from keytab as kudu/127.22.53.254@KRBTEST.COM (short username kudu)
I20260926 07:32:44.454048 4545 webserver.cc:460] Webserver started at http://127.22.53.254:37215/ using document root <none> and password file <none>
I20260926 07:32:44.454662 4545 fs_manager.cc:359] Metadata directory not provided
I20260926 07:32:44.454756 4545 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:32:44.459265 4545 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 07:32:44.461724 4561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:44.463160 4545 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:32:44.463286 4545 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/master-0/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/master-0/wal
uuid: "095dfa56bdf344f6b50d5cc18be740cd"
format_stamp: "Formatted at 2026-09-26 07:32:43 on dist-test-slave-wdjc"
server_key: "fa15f9a965a45d0401f978b68d44e8d6"
server_key_iv: "85062671625e999fae5efde875b2c3d0"
server_key_version: "encryptionkey@0"
I20260926 07:32:44.463944 4545 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:32:44.476780 4545 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:32:44.481406 4545 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:32:44.481631 4545 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:32:44.489904 4545 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.254:35543
I20260926 07:32:44.490214 4613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.254:35543 every 8 connection(s)
I20260926 07:32:44.491564 4545 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
I20260926 07:32:44.497403 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 4545
I20260926 07:32:44.497532 4614 sys_catalog.cc:263] Verifying existing consensus state
I20260926 07:32:44.497967 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.53.193:37097
--local_ip_for_outbound_sockets=127.22.53.193
--tserver_master_addrs=127.22.53.254:35543
--webserver_port=46039
--webserver_interface=127.22.53.193
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.193.keytab
--principal=kudu/127.22.53.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:46315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:32:44.500191 4614 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 095dfa56bdf344f6b50d5cc18be740cd: Bootstrap starting.
I20260926 07:32:44.518633 4614 log.cc:824] T 00000000000000000000000000000000 P 095dfa56bdf344f6b50d5cc18be740cd: Log is configured to *not* fsync() on all Append() calls
I20260926 07:32:44.527817 4614 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 095dfa56bdf344f6b50d5cc18be740cd: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 07:32:44.528265 4614 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 095dfa56bdf344f6b50d5cc18be740cd: Bootstrap complete.
I20260926 07:32:44.532588 4614 raft_consensus.cc:348] T 00000000000000000000000000000000 P 095dfa56bdf344f6b50d5cc18be740cd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "095dfa56bdf344f6b50d5cc18be740cd" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 35543 } }
I20260926 07:32:44.533043 4614 raft_consensus.cc:729] T 00000000000000000000000000000000 P 095dfa56bdf344f6b50d5cc18be740cd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 095dfa56bdf344f6b50d5cc18be740cd, State: Initialized, Role: FOLLOWER
I20260926 07:32:44.533576 4614 consensus_queue.cc:260] T 00000000000000000000000000000000 P 095dfa56bdf344f6b50d5cc18be740cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "095dfa56bdf344f6b50d5cc18be740cd" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 35543 } }
I20260926 07:32:44.533700 4614 raft_consensus.cc:388] T 00000000000000000000000000000000 P 095dfa56bdf344f6b50d5cc18be740cd [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:32:44.533798 4614 raft_consensus.cc:482] T 00000000000000000000000000000000 P 095dfa56bdf344f6b50d5cc18be740cd [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:32:44.533916 4614 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 095dfa56bdf344f6b50d5cc18be740cd [term 1 FOLLOWER]: Advancing to term 2
I20260926 07:32:44.536173 4614 raft_consensus.cc:504] T 00000000000000000000000000000000 P 095dfa56bdf344f6b50d5cc18be740cd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "095dfa56bdf344f6b50d5cc18be740cd" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 35543 } }
I20260926 07:32:44.536582 4614 leader_election.cc:302] T 00000000000000000000000000000000 P 095dfa56bdf344f6b50d5cc18be740cd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 095dfa56bdf344f6b50d5cc18be740cd; no voters:
I20260926 07:32:44.536947 4614 leader_election.cc:288] T 00000000000000000000000000000000 P 095dfa56bdf344f6b50d5cc18be740cd [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 07:32:44.537037 4618 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 095dfa56bdf344f6b50d5cc18be740cd [term 2 FOLLOWER]: Leader election won for term 2
I20260926 07:32:44.537247 4618 raft_consensus.cc:686] T 00000000000000000000000000000000 P 095dfa56bdf344f6b50d5cc18be740cd [term 2 LEADER]: Becoming Leader. State: Replica: 095dfa56bdf344f6b50d5cc18be740cd, State: Running, Role: LEADER
I20260926 07:32:44.537557 4618 consensus_queue.cc:237] T 00000000000000000000000000000000 P 095dfa56bdf344f6b50d5cc18be740cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "095dfa56bdf344f6b50d5cc18be740cd" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 35543 } }
I20260926 07:32:44.538262 4614 sys_catalog.cc:564] T 00000000000000000000000000000000 P 095dfa56bdf344f6b50d5cc18be740cd [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:32:44.538578 4620 sys_catalog.cc:455] T 00000000000000000000000000000000 P 095dfa56bdf344f6b50d5cc18be740cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 095dfa56bdf344f6b50d5cc18be740cd. Latest consensus state: current_term: 2 leader_uuid: "095dfa56bdf344f6b50d5cc18be740cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "095dfa56bdf344f6b50d5cc18be740cd" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 35543 } } }
I20260926 07:32:44.538713 4620 sys_catalog.cc:458] T 00000000000000000000000000000000 P 095dfa56bdf344f6b50d5cc18be740cd [sys.catalog]: This master's current role is: LEADER
I20260926 07:32:44.538949 4619 sys_catalog.cc:455] T 00000000000000000000000000000000 P 095dfa56bdf344f6b50d5cc18be740cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "095dfa56bdf344f6b50d5cc18be740cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "095dfa56bdf344f6b50d5cc18be740cd" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 35543 } } }
I20260926 07:32:44.539053 4619 sys_catalog.cc:458] T 00000000000000000000000000000000 P 095dfa56bdf344f6b50d5cc18be740cd [sys.catalog]: This master's current role is: LEADER
I20260926 07:32:44.539373 4622 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:32:44.544451 4622 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:32:44.545221 4622 catalog_manager.cc:1259] Loaded cluster ID: 96350a859a0d476b9eb3fc198c396466
I20260926 07:32:44.545310 4622 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:32:44.546777 4622 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260926 07:32:44.551738 4622 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260926 07:32:44.558205 4622 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:32:44.559130 4622 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 095dfa56bdf344f6b50d5cc18be740cd: Loaded TSK: 0
I20260926 07:32:44.560433 4622 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 07:32:44.623838 4616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:32:44.624209 4616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:32:44.624306 4616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:32:44.628784 4616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:32:44.628911 4616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:32:44.629012 4616 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.53.193
I20260926 07:32:44.635514 4616 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:46315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.193.keytab
--principal=kudu/127.22.53.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.53.193:37097
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.53.193
--webserver_port=46039
--tserver_master_addrs=127.22.53.254:35543
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4616
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.53.193
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:32:44.637106 4616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:32:44.638775 4616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:32:44.651142 4642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:44.651152 4644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:32:44.651377 4616 server_base.cc:1034] running on GCE node
W20260926 07:32:44.651152 4641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:32:44.652128 4616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:32:44.653081 4616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:32:44.654343 4616 hybrid_clock.cc:648] HybridClock initialized: now 1790407964654300 us; error 58 us; skew 500 ppm
Sep 26 07:32:44 dist-test-slave-wdjc krb5kdc[3997](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407964, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:32:44.658284 4616 init.cc:377] Logged in from keytab as kudu/127.22.53.193@KRBTEST.COM (short username kudu)
I20260926 07:32:44.659595 4616 webserver.cc:460] Webserver started at http://127.22.53.193:46039/ using document root <none> and password file <none>
I20260926 07:32:44.660403 4616 fs_manager.cc:359] Metadata directory not provided
I20260926 07:32:44.660565 4616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:32:44.666239 4616 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 07:32:44.669476 4651 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:44.671182 4616 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 07:32:44.671353 4616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-0/wal
uuid: "84670aa208b740d1bdde3276c1a33264"
format_stamp: "Formatted at 2026-09-26 07:32:43 on dist-test-slave-wdjc"
server_key: "5033f0687a2c951f94f1887fff755d26"
server_key_iv: "9c598874f2d5cacc1ad23cfdada1bf43"
server_key_version: "encryptionkey@0"
I20260926 07:32:44.672137 4616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:32:44.694636 4616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:32:44.699182 4616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:32:44.699417 4616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:32:44.700155 4616 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:32:44.701572 4658 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 07:32:44.706156 4616 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 07:32:44.706264 4616 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260926 07:32:44.706332 4616 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 07:32:44.708484 4616 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 07:32:44.708588 4616 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260926 07:32:44.709007 4658 tablet_bootstrap.cc:492] T 5565e3d3c5ed44f29c9755c4d598d3c8 P 84670aa208b740d1bdde3276c1a33264: Bootstrap starting.
I20260926 07:32:44.719844 4616 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.193:37097
I20260926 07:32:44.719897 4765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.193:37097 every 8 connection(s)
I20260926 07:32:44.721000 4616 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-0/data/info.pb
I20260926 07:32:44.726457 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 4616
I20260926 07:32:44.726981 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.53.194:43777
--local_ip_for_outbound_sockets=127.22.53.194
--tserver_master_addrs=127.22.53.254:35543
--webserver_port=43399
--webserver_interface=127.22.53.194
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.194.keytab
--principal=kudu/127.22.53.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:46315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:32:44.729741 4658 log.cc:824] T 5565e3d3c5ed44f29c9755c4d598d3c8 P 84670aa208b740d1bdde3276c1a33264: Log is configured to *not* fsync() on all Append() calls
Sep 26 07:32:44 dist-test-slave-wdjc krb5kdc[3997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407964, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.193@KRBTEST.COM for kudu/127.22.53.254@KRBTEST.COM
I20260926 07:32:44.738710 4658 tablet_bootstrap.cc:492] T 5565e3d3c5ed44f29c9755c4d598d3c8 P 84670aa208b740d1bdde3276c1a33264: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 07:32:44.739305 4658 tablet_bootstrap.cc:492] T 5565e3d3c5ed44f29c9755c4d598d3c8 P 84670aa208b740d1bdde3276c1a33264: Bootstrap complete.
I20260926 07:32:44.740797 4658 ts_tablet_manager.cc:1397] T 5565e3d3c5ed44f29c9755c4d598d3c8 P 84670aa208b740d1bdde3276c1a33264: Time spent bootstrapping tablet: real 0.032s user 0.021s sys 0.011s
I20260926 07:32:44.743135 4768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:44.723499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.193:41843 (local address 127.22.53.254:35543)
0926 07:32:44.723949 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:44.723968 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:44.724309 (+ 341us) server_negotiation.cc:407] Connection header received
0926 07:32:44.725546 (+ 1237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:44.725563 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:44.726089 (+ 526us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:44.726503 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:44.728044 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:44.729551 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:44.731195 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:44.731990 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:44.732204 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:44.735551 (+ 3347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:44.735561 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:44.735581 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:32:44.735631 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:44.738452 (+ 2821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:44.739376 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:44.739386 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:44.739394 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:44.739598 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:44.740068 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:44.740075 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:44.740079 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:44.740729 (+ 650us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:44.740873 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:44.741680 (+ 807us) server_negotiation.cc:299] Negotiation successful
0926 07:32:44.742038 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":179,"threads_started":1}
I20260926 07:32:44.743165 4769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:44.723424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:35543 (local address 127.22.53.193:41843)
0926 07:32:44.724111 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:44.724165 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 07:32:44.725069 (+ 904us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:32:44.726620 (+ 1551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:32:44.726628 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:32:44.727142 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:32:44.727826 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:44.727851 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:44.729745 (+ 1894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:44.729751 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:44.731045 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:44.731056 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:44.732183 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:44.732187 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:44.732383 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:44.733155 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:32:44.733177 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:32:44.735275 (+ 2098us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:32:44.738628 (+ 3353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:32:44.738635 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:32:44.738649 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:32:44.739149 (+ 500us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:32:44.739730 (+ 581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:32:44.739736 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:32:44.739741 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:32:44.739930 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:32:44.740945 (+ 1015us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:32:44.740953 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:32:44.741279 (+ 326us) client_negotiation.cc:769] Sending connection context
0926 07:32:44.741501 (+ 222us) client_negotiation.cc:240] Negotiation successful
0926 07:32:44.741924 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":167,"threads_started":1}
I20260926 07:32:44.744144 4658 raft_consensus.cc:348] T 5565e3d3c5ed44f29c9755c4d598d3c8 P 84670aa208b740d1bdde3276c1a33264 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84670aa208b740d1bdde3276c1a33264" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 37097 } } peers { permanent_uuid: "bc6ec79640c745879d85774d57594ab4" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 43777 } } peers { permanent_uuid: "2f290d8648c1444dba54d974cfa1a769" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 38919 } }
I20260926 07:32:44.744750 4658 raft_consensus.cc:729] T 5565e3d3c5ed44f29c9755c4d598d3c8 P 84670aa208b740d1bdde3276c1a33264 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84670aa208b740d1bdde3276c1a33264, State: Initialized, Role: FOLLOWER
I20260926 07:32:44.746130 4658 consensus_queue.cc:260] T 5565e3d3c5ed44f29c9755c4d598d3c8 P 84670aa208b740d1bdde3276c1a33264 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84670aa208b740d1bdde3276c1a33264" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 37097 } } peers { permanent_uuid: "bc6ec79640c745879d85774d57594ab4" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 43777 } } peers { permanent_uuid: "2f290d8648c1444dba54d974cfa1a769" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 38919 } }
I20260926 07:32:44.746264 4766 heartbeater.cc:347] Connected to a master server at 127.22.53.254:35543
I20260926 07:32:44.746533 4766 heartbeater.cc:464] Registering TS with master...
I20260926 07:32:44.747165 4766 heartbeater.cc:511] Master 127.22.53.254:35543 requested a full tablet report, sending...
I20260926 07:32:44.747189 4658 ts_tablet_manager.cc:1428] T 5565e3d3c5ed44f29c9755c4d598d3c8 P 84670aa208b740d1bdde3276c1a33264: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260926 07:32:44.749969 4578 ts_manager.cc:194] Registered new tserver with Master: 84670aa208b740d1bdde3276c1a33264 (127.22.53.193:37097)
I20260926 07:32:44.753867 4578 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.53.193@KRBTEST.COM'} at 127.22.53.193:41843
I20260926 07:32:44.756779 4766 heartbeater.cc:503] Master 127.22.53.254:35543 was elected leader, sending a full tablet report...
Sep 26 07:32:44 dist-test-slave-wdjc krb5kdc[3997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407964, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.254@KRBTEST.COM for kudu/127.22.53.193@KRBTEST.COM
I20260926 07:32:44.768257 4773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:44.753336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.193:37097 (local address 127.0.0.1:39564)
0926 07:32:44.754247 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:44.754296 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 07:32:44.754469 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:32:44.755077 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:32:44.755084 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:32:44.757121 (+ 2037us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:32:44.757303 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:44.757313 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:44.758898 (+ 1585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:44.758903 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:44.760088 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:44.760104 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:44.760905 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:44.760910 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:44.761049 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:44.761825 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:32:44.761843 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:32:44.763630 (+ 1787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:32:44.765950 (+ 2320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:32:44.765963 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:32:44.765970 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:32:44.766300 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:32:44.766688 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:32:44.766691 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:32:44.766693 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:32:44.766810 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:32:44.767468 (+ 658us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:32:44.767474 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:32:44.767629 (+ 155us) client_negotiation.cc:769] Sending connection context
0926 07:32:44.767787 (+ 158us) client_negotiation.cc:240] Negotiation successful
0926 07:32:44.768021 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":726,"thread_start_us":137,"threads_started":1}
I20260926 07:32:44.768348 4774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:44.753477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39564 (local address 127.22.53.193:37097)
0926 07:32:44.753967 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:44.753982 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:44.754437 (+ 455us) server_negotiation.cc:407] Connection header received
0926 07:32:44.754651 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:44.754657 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:44.754891 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:44.755054 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:44.757516 (+ 2462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:44.758721 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:44.760245 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:44.760746 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:44.760866 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:44.763849 (+ 2983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:44.763862 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:44.763874 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:32:44.763903 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:44.765803 (+ 1900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:44.766461 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:44.766467 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:44.766468 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:44.766579 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:44.766940 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:44.766943 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:44.766944 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:44.767340 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:44.767497 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:44.767961 (+ 464us) server_negotiation.cc:299] Negotiation successful
0926 07:32:44.768163 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":123,"threads_started":1}
I20260926 07:32:44.769735 4700 tablet_service.cc:1514] Processing DeleteTablet for tablet 5565e3d3c5ed44f29c9755c4d598d3c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:32:44 UTC) from {username='kudu', principal='kudu/127.22.53.254@KRBTEST.COM'} at 127.0.0.1:39564
I20260926 07:32:44.770969 4775 tablet_replica.cc:330] stopping tablet replica
I20260926 07:32:44.771334 4775 raft_consensus.cc:2229] T 5565e3d3c5ed44f29c9755c4d598d3c8 P 84670aa208b740d1bdde3276c1a33264 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:32:44.771633 4775 raft_consensus.cc:2258] T 5565e3d3c5ed44f29c9755c4d598d3c8 P 84670aa208b740d1bdde3276c1a33264 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:32:44.773106 4775 ts_tablet_manager.cc:1905] T 5565e3d3c5ed44f29c9755c4d598d3c8 P 84670aa208b740d1bdde3276c1a33264: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:32:44.777313 4775 ts_tablet_manager.cc:1918] T 5565e3d3c5ed44f29c9755c4d598d3c8 P 84670aa208b740d1bdde3276c1a33264: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:32:44.777422 4775 log.cc:1196] T 5565e3d3c5ed44f29c9755c4d598d3c8 P 84670aa208b740d1bdde3276c1a33264: Deleting WAL directory at /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-0/wal/wals/5565e3d3c5ed44f29c9755c4d598d3c8
I20260926 07:32:44.777736 4775 ts_tablet_manager.cc:1939] T 5565e3d3c5ed44f29c9755c4d598d3c8 P 84670aa208b740d1bdde3276c1a33264: Deleting consensus metadata
I20260926 07:32:44.779623 4563 catalog_manager.cc:4841] TS 84670aa208b740d1bdde3276c1a33264 (127.22.53.193:37097): tablet 5565e3d3c5ed44f29c9755c4d598d3c8 (table test-table [id=74398147aff446a0bf197dbcb3f49c79]) successfully deleted
W20260926 07:32:44.860340 4770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:32:44.860752 4770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:32:44.860855 4770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:32:44.865154 4770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:32:44.865257 4770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:32:44.865352 4770 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.53.194
I20260926 07:32:44.871101 4770 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:46315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.194.keytab
--principal=kudu/127.22.53.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.53.194:43777
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.53.194
--webserver_port=43399
--tserver_master_addrs=127.22.53.254:35543
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4770
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.53.194
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:32:44.872318 4770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:32:44.873441 4770 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:32:44.884428 4780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:44.884449 4783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:44.884428 4781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:32:44.884768 4770 server_base.cc:1034] running on GCE node
I20260926 07:32:44.885339 4770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:32:44.886207 4770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:32:44.887430 4770 hybrid_clock.cc:648] HybridClock initialized: now 1790407964887386 us; error 57 us; skew 500 ppm
Sep 26 07:32:44 dist-test-slave-wdjc krb5kdc[3997](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407964, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:32:44.890803 4770 init.cc:377] Logged in from keytab as kudu/127.22.53.194@KRBTEST.COM (short username kudu)
I20260926 07:32:44.891804 4770 webserver.cc:460] Webserver started at http://127.22.53.194:43399/ using document root <none> and password file <none>
I20260926 07:32:44.892421 4770 fs_manager.cc:359] Metadata directory not provided
I20260926 07:32:44.892534 4770 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:32:44.896822 4770 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260926 07:32:44.899235 4790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:44.900799 4770 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:32:44.900964 4770 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-1/wal
uuid: "bc6ec79640c745879d85774d57594ab4"
format_stamp: "Formatted at 2026-09-26 07:32:43 on dist-test-slave-wdjc"
server_key: "ca7f2bc93ac94e8a8f22a2e9860eb593"
server_key_iv: "b22cde5e5514a23f3e64d49aef11e2fe"
server_key_version: "encryptionkey@0"
I20260926 07:32:44.901589 4770 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:32:44.912065 4770 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:32:44.915390 4770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:32:44.915624 4770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:32:44.916268 4770 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:32:44.917285 4770 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:32:44.917362 4770 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:44.917430 4770 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:32:44.917481 4770 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:44.927597 4770 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.194:43777
I20260926 07:32:44.927637 4903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.194:43777 every 8 connection(s)
I20260926 07:32:44.928699 4770 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-1/data/info.pb
I20260926 07:32:44.936297 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 4770
I20260926 07:32:44.936818 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.53.195:38919
--local_ip_for_outbound_sockets=127.22.53.195
--tserver_master_addrs=127.22.53.254:35543
--webserver_port=41915
--webserver_interface=127.22.53.195
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.195.keytab
--principal=kudu/127.22.53.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:46315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 07:32:44 dist-test-slave-wdjc krb5kdc[3997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407964, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.194@KRBTEST.COM for kudu/127.22.53.254@KRBTEST.COM
I20260926 07:32:44.945725 4768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:44.930765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.194:39573 (local address 127.22.53.254:35543)
0926 07:32:44.930946 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:44.930950 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:44.931812 (+ 862us) server_negotiation.cc:407] Connection header received
0926 07:32:44.933308 (+ 1496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:44.933310 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:44.933366 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:44.933451 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:44.934740 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:44.935306 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:44.936821 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:44.937174 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:44.937276 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:44.940788 (+ 3512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:44.940796 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:44.940802 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:32:44.940841 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:44.942805 (+ 1964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:44.943528 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:44.943534 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:44.943538 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:44.943655 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:44.944136 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:44.944143 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:44.944147 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:44.944453 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:44.944608 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:44.945284 (+ 676us) server_negotiation.cc:299] Negotiation successful
0926 07:32:44.945473 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 07:32:44.946434 4906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:44.931154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:35543 (local address 127.22.53.194:39573)
0926 07:32:44.931634 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:44.931678 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 07:32:44.933065 (+ 1387us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:32:44.933642 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:32:44.933649 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:32:44.934055 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:32:44.934568 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:44.934582 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:44.935428 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:44.935430 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:44.936670 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:44.936685 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:44.937364 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:44.937372 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:44.937626 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:44.938431 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:32:44.938460 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:32:44.940511 (+ 2051us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:32:44.942974 (+ 2463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:32:44.942981 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:32:44.942996 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:32:44.943366 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:32:44.943806 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:32:44.943809 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:32:44.943810 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:32:44.944007 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:32:44.944667 (+ 660us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:32:44.944673 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:32:44.944948 (+ 275us) client_negotiation.cc:769] Sending connection context
0926 07:32:44.945166 (+ 218us) client_negotiation.cc:240] Negotiation successful
0926 07:32:44.945468 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":127,"threads_started":1}
I20260926 07:32:44.947896 4904 heartbeater.cc:347] Connected to a master server at 127.22.53.254:35543
I20260926 07:32:44.948220 4904 heartbeater.cc:464] Registering TS with master...
I20260926 07:32:44.948992 4904 heartbeater.cc:511] Master 127.22.53.254:35543 requested a full tablet report, sending...
I20260926 07:32:44.950326 4578 ts_manager.cc:194] Registered new tserver with Master: bc6ec79640c745879d85774d57594ab4 (127.22.53.194:43777)
I20260926 07:32:44.951124 4578 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.53.194@KRBTEST.COM'} at 127.22.53.194:39573
W20260926 07:32:45.055667 4907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:32:45.055971 4907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:32:45.056066 4907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:32:45.060391 4907 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:32:45.060465 4907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:32:45.060601 4907 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.53.195
I20260926 07:32:45.066366 4907 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:46315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.195.keytab
--principal=kudu/127.22.53.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.53.195:38919
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.53.195
--webserver_port=41915
--tserver_master_addrs=127.22.53.254:35543
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4907
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.53.195
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:32:45.067617 4907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:32:45.068779 4907 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:32:45.079066 4915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:45.079066 4913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:45.079103 4912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:32:45.079324 4907 server_base.cc:1034] running on GCE node
I20260926 07:32:45.079921 4907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:32:45.080758 4907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:32:45.081957 4907 hybrid_clock.cc:648] HybridClock initialized: now 1790407965081941 us; error 30 us; skew 500 ppm
Sep 26 07:32:45 dist-test-slave-wdjc krb5kdc[3997](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407965, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:32:45.085049 4907 init.cc:377] Logged in from keytab as kudu/127.22.53.195@KRBTEST.COM (short username kudu)
I20260926 07:32:45.086319 4907 webserver.cc:460] Webserver started at http://127.22.53.195:41915/ using document root <none> and password file <none>
I20260926 07:32:45.087050 4907 fs_manager.cc:359] Metadata directory not provided
I20260926 07:32:45.087158 4907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:32:45.091765 4907 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 07:32:45.094350 4922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:45.095842 4907 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 07:32:45.096001 4907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-2/wal
uuid: "2f290d8648c1444dba54d974cfa1a769"
format_stamp: "Formatted at 2026-09-26 07:32:43 on dist-test-slave-wdjc"
server_key: "481e228797e50ef9032d222ff42d3a8b"
server_key_iv: "3405b1c091992969b8ffbec0bbba738b"
server_key_version: "encryptionkey@0"
I20260926 07:32:45.096652 4907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:32:45.119886 4907 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:32:45.123368 4907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:32:45.123590 4907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:32:45.124251 4907 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:32:45.125212 4907 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:32:45.125269 4907 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:45.125339 4907 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:32:45.125372 4907 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:45.135381 4907 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.195:38919
I20260926 07:32:45.135402 5035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.195:38919 every 8 connection(s)
I20260926 07:32:45.136512 4907 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-2/data/info.pb
I20260926 07:32:45.145439 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 4907
Sep 26 07:32:45 dist-test-slave-wdjc krb5kdc[3997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407965, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.195@KRBTEST.COM for kudu/127.22.53.254@KRBTEST.COM
I20260926 07:32:45.154739 4768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:45.138860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.195:34373 (local address 127.22.53.254:35543)
0926 07:32:45.139123 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:45.139127 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:45.140051 (+ 924us) server_negotiation.cc:407] Connection header received
0926 07:32:45.141733 (+ 1682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:45.141736 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:45.141806 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:45.141902 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:45.143509 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:45.144454 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:45.146295 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:45.146642 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:45.146770 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:45.150494 (+ 3724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:45.150498 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:45.150500 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:32:45.150529 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:45.152034 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:45.152913 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:45.152919 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:45.152921 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:45.153012 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:45.153452 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:45.153455 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:45.153457 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:45.153673 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:45.153795 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:45.154403 (+ 608us) server_negotiation.cc:299] Negotiation successful
0926 07:32:45.154554 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"spinlock_wait_cycles":4352}
I20260926 07:32:45.155905 5040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:45.146313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40168 (local address 127.22.53.254:35543)
0926 07:32:45.146616 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:45.146619 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:45.146744 (+ 125us) server_negotiation.cc:407] Connection header received
0926 07:32:45.146854 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:45.146858 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:45.146912 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:45.147015 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:45.147938 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:45.148771 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:45.150198 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:45.150446 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:45.150571 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:45.152198 (+ 1627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:45.152201 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:45.152203 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:32:45.152227 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:45.153839 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:45.154322 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:45.154328 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:45.154330 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:45.154435 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:45.154755 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:45.154759 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:45.154761 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:45.155037 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:45.155264 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:45.155521 (+ 257us) server_negotiation.cc:299] Negotiation successful
0926 07:32:45.155669 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"spinlock_wait_cycles":2944,"thread_start_us":82,"threads_started":1}
I20260926 07:32:45.155951 5038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:45.139313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:35543 (local address 127.22.53.195:34373)
0926 07:32:45.139884 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:45.139929 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 07:32:45.141423 (+ 1494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:32:45.142172 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:32:45.142181 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:32:45.142634 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:32:45.143195 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:45.143210 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:45.144736 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:45.144739 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:45.146132 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:45.146148 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:45.146876 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:45.146881 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:45.147145 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:45.147953 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:32:45.147977 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:32:45.150281 (+ 2304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:32:45.152238 (+ 1957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:32:45.152246 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:32:45.152269 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:32:45.152765 (+ 496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:32:45.153133 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:32:45.153136 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:32:45.153138 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:32:45.153317 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:32:45.153795 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:32:45.153804 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:32:45.154113 (+ 309us) client_negotiation.cc:769] Sending connection context
0926 07:32:45.154328 (+ 215us) client_negotiation.cc:240] Negotiation successful
0926 07:32:45.154751 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":146,"threads_started":1}
I20260926 07:32:45.157546 5036 heartbeater.cc:347] Connected to a master server at 127.22.53.254:35543
I20260926 07:32:45.157964 5036 heartbeater.cc:464] Registering TS with master...
I20260926 07:32:45.158623 5036 heartbeater.cc:511] Master 127.22.53.254:35543 requested a full tablet report, sending...
I20260926 07:32:45.159945 4578 ts_manager.cc:194] Registered new tserver with Master: 2f290d8648c1444dba54d974cfa1a769 (127.22.53.195:38919)
I20260926 07:32:45.160840 4578 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.53.195@KRBTEST.COM'} at 127.22.53.195:34373
I20260926 07:32:45.170784 22743 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:32:45.176323 5040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:45.171535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40184 (local address 127.22.53.254:35543)
0926 07:32:45.171700 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:45.171705 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:45.171957 (+ 252us) server_negotiation.cc:407] Connection header received
0926 07:32:45.172105 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:45.172107 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:45.172251 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:45.172361 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:32:45.172643 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:45.173185 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:45.174082 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:45.174336 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:45.174434 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:45.175115 (+ 681us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:32:45.175759 (+ 644us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:32:45.176071 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:45.176111 (+ 40us) server_negotiation.cc:299] Negotiation successful
0926 07:32:45.176191 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 07:32:45.177289 4578 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:40184:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:32:45.181515 4578 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:32:45.194113 4700 tablet_service.cc:1467] Processing CreateTablet for tablet 72c66b98ff0249268a5aa0a60e70851d (DEFAULT_TABLE table=test-table [id=d144eadd58a448fabc3973b516392202]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:32:45.194926 4700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72c66b98ff0249268a5aa0a60e70851d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:32:45.198040 4773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:45.192381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.195:38919 (local address 127.0.0.1:57956)
0926 07:32:45.192600 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:45.192617 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 07:32:45.192710 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:32:45.193717 (+ 1007us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:32:45.193721 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:32:45.193755 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:32:45.193887 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:45.193897 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:45.195421 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:45.195426 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:45.196315 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:45.196326 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:45.197545 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:45.197548 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:45.197670 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:45.197764 (+ 94us) client_negotiation.cc:769] Sending connection context
0926 07:32:45.197824 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 07:32:45.197922 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98}
I20260926 07:32:45.198506 4658 tablet_bootstrap.cc:492] T 72c66b98ff0249268a5aa0a60e70851d P 84670aa208b740d1bdde3276c1a33264: Bootstrap starting.
I20260926 07:32:45.198696 5046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:45.192735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.194:43777 (local address 127.0.0.1:48390)
0926 07:32:45.193145 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:45.193161 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:32:45.193263 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:32:45.193990 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:32:45.193993 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:32:45.194011 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:32:45.194103 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:45.194109 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:45.195586 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:45.195590 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:45.196754 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:45.196767 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:45.198093 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:45.198097 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:45.198328 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:45.198361 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 07:32:45.198482 (+ 121us) client_negotiation.cc:240] Negotiation successful
0926 07:32:45.198575 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":144,"threads_started":1}
I20260926 07:32:45.198990 5045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:45.192633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57956 (local address 127.22.53.195:38919)
0926 07:32:45.193059 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:45.193073 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:45.193100 (+ 27us) server_negotiation.cc:407] Connection header received
0926 07:32:45.193204 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:45.193212 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:45.193472 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:45.193741 (+ 269us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:32:45.194029 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:45.195262 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:45.196469 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:45.197409 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:45.198367 (+ 958us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:45.198533 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:45.198583 (+ 50us) server_negotiation.cc:299] Negotiation successful
0926 07:32:45.198701 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":181,"threads_started":1}
I20260926 07:32:45.199055 5047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:45.192913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48390 (local address 127.22.53.194:43777)
0926 07:32:45.193356 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:45.193369 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:45.193397 (+ 28us) server_negotiation.cc:407] Connection header received
0926 07:32:45.193461 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:45.193465 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:45.193659 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:45.193816 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:32:45.194261 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:45.195398 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:45.196908 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:45.197736 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:45.197849 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:45.198021 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:45.198541 (+ 520us) server_negotiation.cc:299] Negotiation successful
0926 07:32:45.198709 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":119,"threads_started":1}
I20260926 07:32:45.199630 4658 tablet_bootstrap.cc:654] T 72c66b98ff0249268a5aa0a60e70851d P 84670aa208b740d1bdde3276c1a33264: Neither blocks nor log segments found. Creating new log.
I20260926 07:32:45.201511 4658 tablet_bootstrap.cc:492] T 72c66b98ff0249268a5aa0a60e70851d P 84670aa208b740d1bdde3276c1a33264: No bootstrap required, opened a new log
I20260926 07:32:45.201637 4658 ts_tablet_manager.cc:1397] T 72c66b98ff0249268a5aa0a60e70851d P 84670aa208b740d1bdde3276c1a33264: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 07:32:45.202180 4658 raft_consensus.cc:348] T 72c66b98ff0249268a5aa0a60e70851d P 84670aa208b740d1bdde3276c1a33264 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f290d8648c1444dba54d974cfa1a769" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 38919 } } peers { permanent_uuid: "bc6ec79640c745879d85774d57594ab4" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 43777 } } peers { permanent_uuid: "84670aa208b740d1bdde3276c1a33264" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 37097 } }
I20260926 07:32:45.202327 4658 raft_consensus.cc:374] T 72c66b98ff0249268a5aa0a60e70851d P 84670aa208b740d1bdde3276c1a33264 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:32:45.202366 4658 raft_consensus.cc:729] T 72c66b98ff0249268a5aa0a60e70851d P 84670aa208b740d1bdde3276c1a33264 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84670aa208b740d1bdde3276c1a33264, State: Initialized, Role: FOLLOWER
I20260926 07:32:45.202557 4658 consensus_queue.cc:260] T 72c66b98ff0249268a5aa0a60e70851d P 84670aa208b740d1bdde3276c1a33264 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f290d8648c1444dba54d974cfa1a769" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 38919 } } peers { permanent_uuid: "bc6ec79640c745879d85774d57594ab4" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 43777 } } peers { permanent_uuid: "84670aa208b740d1bdde3276c1a33264" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 37097 } }
I20260926 07:32:45.202597 4838 tablet_service.cc:1467] Processing CreateTablet for tablet 72c66b98ff0249268a5aa0a60e70851d (DEFAULT_TABLE table=test-table [id=d144eadd58a448fabc3973b516392202]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:32:45.202585 4970 tablet_service.cc:1467] Processing CreateTablet for tablet 72c66b98ff0249268a5aa0a60e70851d (DEFAULT_TABLE table=test-table [id=d144eadd58a448fabc3973b516392202]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:32:45.203397 4658 ts_tablet_manager.cc:1428] T 72c66b98ff0249268a5aa0a60e70851d P 84670aa208b740d1bdde3276c1a33264: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 07:32:45.203822 4838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72c66b98ff0249268a5aa0a60e70851d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:32:45.203840 4970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72c66b98ff0249268a5aa0a60e70851d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:32:45.212204 5049 tablet_bootstrap.cc:492] T 72c66b98ff0249268a5aa0a60e70851d P bc6ec79640c745879d85774d57594ab4: Bootstrap starting.
I20260926 07:32:45.212316 5050 tablet_bootstrap.cc:492] T 72c66b98ff0249268a5aa0a60e70851d P 2f290d8648c1444dba54d974cfa1a769: Bootstrap starting.
I20260926 07:32:45.215108 5049 tablet_bootstrap.cc:654] T 72c66b98ff0249268a5aa0a60e70851d P bc6ec79640c745879d85774d57594ab4: Neither blocks nor log segments found. Creating new log.
I20260926 07:32:45.215332 5050 tablet_bootstrap.cc:654] T 72c66b98ff0249268a5aa0a60e70851d P 2f290d8648c1444dba54d974cfa1a769: Neither blocks nor log segments found. Creating new log.
I20260926 07:32:45.216075 5049 log.cc:824] T 72c66b98ff0249268a5aa0a60e70851d P bc6ec79640c745879d85774d57594ab4: Log is configured to *not* fsync() on all Append() calls
I20260926 07:32:45.216100 5050 log.cc:824] T 72c66b98ff0249268a5aa0a60e70851d P 2f290d8648c1444dba54d974cfa1a769: Log is configured to *not* fsync() on all Append() calls
I20260926 07:32:45.218590 5050 tablet_bootstrap.cc:492] T 72c66b98ff0249268a5aa0a60e70851d P 2f290d8648c1444dba54d974cfa1a769: No bootstrap required, opened a new log
I20260926 07:32:45.218580 5049 tablet_bootstrap.cc:492] T 72c66b98ff0249268a5aa0a60e70851d P bc6ec79640c745879d85774d57594ab4: No bootstrap required, opened a new log
I20260926 07:32:45.218865 5050 ts_tablet_manager.cc:1397] T 72c66b98ff0249268a5aa0a60e70851d P 2f290d8648c1444dba54d974cfa1a769: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260926 07:32:45.218868 5049 ts_tablet_manager.cc:1397] T 72c66b98ff0249268a5aa0a60e70851d P bc6ec79640c745879d85774d57594ab4: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
W20260926 07:32:45.222354 4767 tablet.cc:2369] T 72c66b98ff0249268a5aa0a60e70851d P 84670aa208b740d1bdde3276c1a33264: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:32:45.222333 5050 raft_consensus.cc:348] T 72c66b98ff0249268a5aa0a60e70851d P 2f290d8648c1444dba54d974cfa1a769 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f290d8648c1444dba54d974cfa1a769" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 38919 } } peers { permanent_uuid: "bc6ec79640c745879d85774d57594ab4" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 43777 } } peers { permanent_uuid: "84670aa208b740d1bdde3276c1a33264" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 37097 } }
I20260926 07:32:45.222678 5050 raft_consensus.cc:374] T 72c66b98ff0249268a5aa0a60e70851d P 2f290d8648c1444dba54d974cfa1a769 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:32:45.222755 5050 raft_consensus.cc:729] T 72c66b98ff0249268a5aa0a60e70851d P 2f290d8648c1444dba54d974cfa1a769 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f290d8648c1444dba54d974cfa1a769, State: Initialized, Role: FOLLOWER
I20260926 07:32:45.223287 5050 consensus_queue.cc:260] T 72c66b98ff0249268a5aa0a60e70851d P 2f290d8648c1444dba54d974cfa1a769 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f290d8648c1444dba54d974cfa1a769" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 38919 } } peers { permanent_uuid: "bc6ec79640c745879d85774d57594ab4" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 43777 } } peers { permanent_uuid: "84670aa208b740d1bdde3276c1a33264" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 37097 } }
I20260926 07:32:45.223287 5049 raft_consensus.cc:348] T 72c66b98ff0249268a5aa0a60e70851d P bc6ec79640c745879d85774d57594ab4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f290d8648c1444dba54d974cfa1a769" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 38919 } } peers { permanent_uuid: "bc6ec79640c745879d85774d57594ab4" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 43777 } } peers { permanent_uuid: "84670aa208b740d1bdde3276c1a33264" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 37097 } }
I20260926 07:32:45.223960 5049 raft_consensus.cc:374] T 72c66b98ff0249268a5aa0a60e70851d P bc6ec79640c745879d85774d57594ab4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:32:45.224057 5049 raft_consensus.cc:729] T 72c66b98ff0249268a5aa0a60e70851d P bc6ec79640c745879d85774d57594ab4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc6ec79640c745879d85774d57594ab4, State: Initialized, Role: FOLLOWER
I20260926 07:32:45.224519 5036 heartbeater.cc:503] Master 127.22.53.254:35543 was elected leader, sending a full tablet report...
I20260926 07:32:45.224710 5050 ts_tablet_manager.cc:1428] T 72c66b98ff0249268a5aa0a60e70851d P 2f290d8648c1444dba54d974cfa1a769: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260926 07:32:45.225158 5049 consensus_queue.cc:260] T 72c66b98ff0249268a5aa0a60e70851d P bc6ec79640c745879d85774d57594ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f290d8648c1444dba54d974cfa1a769" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 38919 } } peers { permanent_uuid: "bc6ec79640c745879d85774d57594ab4" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 43777 } } peers { permanent_uuid: "84670aa208b740d1bdde3276c1a33264" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 37097 } }
I20260926 07:32:45.226609 5049 ts_tablet_manager.cc:1428] T 72c66b98ff0249268a5aa0a60e70851d P bc6ec79640c745879d85774d57594ab4: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260926 07:32:45.227267 4904 heartbeater.cc:503] Master 127.22.53.254:35543 was elected leader, sending a full tablet report...
I20260926 07:32:45.276196 5054 raft_consensus.cc:482] T 72c66b98ff0249268a5aa0a60e70851d P bc6ec79640c745879d85774d57594ab4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:32:45.276435 5054 raft_consensus.cc:504] T 72c66b98ff0249268a5aa0a60e70851d P bc6ec79640c745879d85774d57594ab4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f290d8648c1444dba54d974cfa1a769" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 38919 } } peers { permanent_uuid: "bc6ec79640c745879d85774d57594ab4" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 43777 } } peers { permanent_uuid: "84670aa208b740d1bdde3276c1a33264" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 37097 } }
I20260926 07:32:45.277818 5054 leader_election.cc:288] T 72c66b98ff0249268a5aa0a60e70851d P bc6ec79640c745879d85774d57594ab4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2f290d8648c1444dba54d974cfa1a769 (127.22.53.195:38919), 84670aa208b740d1bdde3276c1a33264 (127.22.53.193:37097)
I20260926 07:32:45.283218 5045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:45.278190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.194:60967 (local address 127.22.53.195:38919)
0926 07:32:45.278337 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:45.278344 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:45.278363 (+ 19us) server_negotiation.cc:407] Connection header received
0926 07:32:45.278813 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:45.278817 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:45.278899 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:45.279004 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:32:45.279341 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:45.280043 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:45.281634 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:45.282311 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:45.282454 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:45.282529 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:45.282972 (+ 443us) server_negotiation.cc:299] Negotiation successful
0926 07:32:45.283056 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 07:32:45.283211 5055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:45.278158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.194:36455 (local address 127.22.53.193:37097)
0926 07:32:45.278463 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:45.278469 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:45.278909 (+ 440us) server_negotiation.cc:407] Connection header received
0926 07:32:45.279037 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:45.279040 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:45.279130 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:45.279241 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:32:45.279728 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:45.280364 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:45.281791 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:45.282460 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:45.282585 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:45.282696 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:45.282972 (+ 276us) server_negotiation.cc:299] Negotiation successful
0926 07:32:45.283056 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":111,"threads_started":1}
I20260926 07:32:45.283237 4906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:45.278080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.195:38919 (local address 127.22.53.194:60967)
0926 07:32:45.278259 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:45.278278 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 07:32:45.278381 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:32:45.279032 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:32:45.279035 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:32:45.279048 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:32:45.279187 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:45.279192 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:45.280168 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:45.280170 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:45.281483 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:45.281495 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:45.282562 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:45.282566 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:45.282814 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:45.282853 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 07:32:45.282941 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 07:32:45.283056 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"mutex_wait_us":94}
I20260926 07:32:45.283246 5056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:45.278080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.193:37097 (local address 127.22.53.194:36455)
0926 07:32:45.278833 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:45.278845 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 07:32:45.278928 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:32:45.279269 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:32:45.279272 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:32:45.279295 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:32:45.279592 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:45.279599 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:45.280471 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:45.280475 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:45.281648 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:45.281662 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:45.282576 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:45.282579 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:45.282787 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:45.282850 (+ 63us) client_negotiation.cc:769] Sending connection context
0926 07:32:45.282941 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 07:32:45.283056 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":685,"mutex_wait_us":459,"thread_start_us":166,"threads_started":1}
I20260926 07:32:45.284171 4990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72c66b98ff0249268a5aa0a60e70851d" candidate_uuid: "bc6ec79640c745879d85774d57594ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f290d8648c1444dba54d974cfa1a769" is_pre_election: true
I20260926 07:32:45.284207 4720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72c66b98ff0249268a5aa0a60e70851d" candidate_uuid: "bc6ec79640c745879d85774d57594ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84670aa208b740d1bdde3276c1a33264" is_pre_election: true
I20260926 07:32:45.284541 4990 raft_consensus.cc:2454] T 72c66b98ff0249268a5aa0a60e70851d P 2f290d8648c1444dba54d974cfa1a769 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc6ec79640c745879d85774d57594ab4 in term 0.
I20260926 07:32:45.284629 4720 raft_consensus.cc:2454] T 72c66b98ff0249268a5aa0a60e70851d P 84670aa208b740d1bdde3276c1a33264 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc6ec79640c745879d85774d57594ab4 in term 0.
I20260926 07:32:45.285202 4792 leader_election.cc:302] T 72c66b98ff0249268a5aa0a60e70851d P bc6ec79640c745879d85774d57594ab4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2f290d8648c1444dba54d974cfa1a769, 84670aa208b740d1bdde3276c1a33264, bc6ec79640c745879d85774d57594ab4; no voters:
I20260926 07:32:45.285547 5054 raft_consensus.cc:2781] T 72c66b98ff0249268a5aa0a60e70851d P bc6ec79640c745879d85774d57594ab4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:32:45.285635 5054 raft_consensus.cc:482] T 72c66b98ff0249268a5aa0a60e70851d P bc6ec79640c745879d85774d57594ab4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:32:45.285668 5054 raft_consensus.cc:3037] T 72c66b98ff0249268a5aa0a60e70851d P bc6ec79640c745879d85774d57594ab4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:32:45.286913 5054 raft_consensus.cc:504] T 72c66b98ff0249268a5aa0a60e70851d P bc6ec79640c745879d85774d57594ab4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f290d8648c1444dba54d974cfa1a769" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 38919 } } peers { permanent_uuid: "bc6ec79640c745879d85774d57594ab4" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 43777 } } peers { permanent_uuid: "84670aa208b740d1bdde3276c1a33264" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 37097 } }
I20260926 07:32:45.287444 5054 leader_election.cc:288] T 72c66b98ff0249268a5aa0a60e70851d P bc6ec79640c745879d85774d57594ab4 [CANDIDATE]: Term 1 election: Requested vote from peers 2f290d8648c1444dba54d974cfa1a769 (127.22.53.195:38919), 84670aa208b740d1bdde3276c1a33264 (127.22.53.193:37097)
I20260926 07:32:45.287956 4990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72c66b98ff0249268a5aa0a60e70851d" candidate_uuid: "bc6ec79640c745879d85774d57594ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f290d8648c1444dba54d974cfa1a769"
I20260926 07:32:45.287956 4720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72c66b98ff0249268a5aa0a60e70851d" candidate_uuid: "bc6ec79640c745879d85774d57594ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84670aa208b740d1bdde3276c1a33264"
I20260926 07:32:45.288123 4720 raft_consensus.cc:3037] T 72c66b98ff0249268a5aa0a60e70851d P 84670aa208b740d1bdde3276c1a33264 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:32:45.288123 4990 raft_consensus.cc:3037] T 72c66b98ff0249268a5aa0a60e70851d P 2f290d8648c1444dba54d974cfa1a769 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:32:45.289609 4720 raft_consensus.cc:2454] T 72c66b98ff0249268a5aa0a60e70851d P 84670aa208b740d1bdde3276c1a33264 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc6ec79640c745879d85774d57594ab4 in term 1.
I20260926 07:32:45.289609 4990 raft_consensus.cc:2454] T 72c66b98ff0249268a5aa0a60e70851d P 2f290d8648c1444dba54d974cfa1a769 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc6ec79640c745879d85774d57594ab4 in term 1.
I20260926 07:32:45.290061 4791 leader_election.cc:302] T 72c66b98ff0249268a5aa0a60e70851d P bc6ec79640c745879d85774d57594ab4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f290d8648c1444dba54d974cfa1a769, bc6ec79640c745879d85774d57594ab4; no voters:
I20260926 07:32:45.290403 5054 raft_consensus.cc:2781] T 72c66b98ff0249268a5aa0a60e70851d P bc6ec79640c745879d85774d57594ab4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:32:45.290822 5054 raft_consensus.cc:686] T 72c66b98ff0249268a5aa0a60e70851d P bc6ec79640c745879d85774d57594ab4 [term 1 LEADER]: Becoming Leader. State: Replica: bc6ec79640c745879d85774d57594ab4, State: Running, Role: LEADER
I20260926 07:32:45.291411 5054 consensus_queue.cc:237] T 72c66b98ff0249268a5aa0a60e70851d P bc6ec79640c745879d85774d57594ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f290d8648c1444dba54d974cfa1a769" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 38919 } } peers { permanent_uuid: "bc6ec79640c745879d85774d57594ab4" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 43777 } } peers { permanent_uuid: "84670aa208b740d1bdde3276c1a33264" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 37097 } }
I20260926 07:32:45.295493 4577 catalog_manager.cc:5495] T 72c66b98ff0249268a5aa0a60e70851d P bc6ec79640c745879d85774d57594ab4 reported cstate change: term changed from 0 to 1, leader changed from <none> to bc6ec79640c745879d85774d57594ab4 (127.22.53.194). New cstate: current_term: 1 leader_uuid: "bc6ec79640c745879d85774d57594ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f290d8648c1444dba54d974cfa1a769" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 38919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc6ec79640c745879d85774d57594ab4" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 43777 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84670aa208b740d1bdde3276c1a33264" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 37097 } health_report { overall_health: UNKNOWN } } }
I20260926 07:32:45.329073 5047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:45.325236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48398 (local address 127.22.53.194:43777)
0926 07:32:45.325413 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:45.325419 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:45.325437 (+ 18us) server_negotiation.cc:407] Connection header received
0926 07:32:45.325524 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:45.325527 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:45.325587 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:45.325709 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:32:45.325973 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:45.326495 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:45.327352 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:45.327576 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:45.327700 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:45.328048 (+ 348us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:32:45.328602 (+ 554us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:32:45.328747 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:45.328813 (+ 66us) server_negotiation.cc:299] Negotiation successful
0926 07:32:45.328882 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 07:32:45.336226 4720 raft_consensus.cc:1261] T 72c66b98ff0249268a5aa0a60e70851d P 84670aa208b740d1bdde3276c1a33264 [term 1 FOLLOWER]: Refusing update from remote peer bc6ec79640c745879d85774d57594ab4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:32:45.336246 4990 raft_consensus.cc:1261] T 72c66b98ff0249268a5aa0a60e70851d P 2f290d8648c1444dba54d974cfa1a769 [term 1 FOLLOWER]: Refusing update from remote peer bc6ec79640c745879d85774d57594ab4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:32:45.337152 5054 consensus_queue.cc:1035] T 72c66b98ff0249268a5aa0a60e70851d P bc6ec79640c745879d85774d57594ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f290d8648c1444dba54d974cfa1a769" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 38919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:32:45.337445 5057 consensus_queue.cc:1035] T 72c66b98ff0249268a5aa0a60e70851d P bc6ec79640c745879d85774d57594ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84670aa208b740d1bdde3276c1a33264" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 37097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:32:45.345825 5059 mvcc.cc:205] Tried to move back new op lower bound from 7333511026007269376 to 7333511025837744128. Current Snapshot: MvccSnapshot[applied={T|T < 7333511026007269376 or (T in {7333511026007269376})}]
I20260926 07:32:45.348114 5061 mvcc.cc:205] Tried to move back new op lower bound from 7333511026007269376 to 7333511025837744128. Current Snapshot: MvccSnapshot[applied={T|T < 7333511026007269376 or (T in {7333511026007269376})}]
I20260926 07:32:45.348569 5060 mvcc.cc:205] Tried to move back new op lower bound from 7333511026007269376 to 7333511025837744128. Current Snapshot: MvccSnapshot[applied={T|T < 7333511026007269376}]
I20260926 07:32:45.355026 4578 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:40184:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:32:45.355235 4578 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:40184:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:32:45.358250 4578 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 095dfa56bdf344f6b50d5cc18be740cd: Sending DeleteTablet for 3 replicas of tablet 72c66b98ff0249268a5aa0a60e70851d
I20260926 07:32:45.359253 4970 tablet_service.cc:1514] Processing DeleteTablet for tablet 72c66b98ff0249268a5aa0a60e70851d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:32:45 UTC) from {username='kudu', principal='kudu/127.22.53.254@KRBTEST.COM'} at 127.0.0.1:57956
I20260926 07:32:45.359383 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 4616
I20260926 07:32:45.359377 4838 tablet_service.cc:1514] Processing DeleteTablet for tablet 72c66b98ff0249268a5aa0a60e70851d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:32:45 UTC) from {username='kudu', principal='kudu/127.22.53.254@KRBTEST.COM'} at 127.0.0.1:48390
I20260926 07:32:45.360146 5069 tablet_replica.cc:330] T 72c66b98ff0249268a5aa0a60e70851d P bc6ec79640c745879d85774d57594ab4: stopping tablet replica
I20260926 07:32:45.360577 5069 raft_consensus.cc:2229] T 72c66b98ff0249268a5aa0a60e70851d P bc6ec79640c745879d85774d57594ab4 [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:32:45.361012 5069 raft_consensus.cc:2258] T 72c66b98ff0249268a5aa0a60e70851d P bc6ec79640c745879d85774d57594ab4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:32:45.361302 5068 tablet_replica.cc:330] T 72c66b98ff0249268a5aa0a60e70851d P 2f290d8648c1444dba54d974cfa1a769: stopping tablet replica
I20260926 07:32:45.361618 5068 raft_consensus.cc:2229] T 72c66b98ff0249268a5aa0a60e70851d P 2f290d8648c1444dba54d974cfa1a769 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:32:45.362026 5068 raft_consensus.cc:2258] T 72c66b98ff0249268a5aa0a60e70851d P 2f290d8648c1444dba54d974cfa1a769 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:32:45.362918 5069 ts_tablet_manager.cc:1905] T 72c66b98ff0249268a5aa0a60e70851d P bc6ec79640c745879d85774d57594ab4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:32:45.363525 5068 ts_tablet_manager.cc:1905] T 72c66b98ff0249268a5aa0a60e70851d P 2f290d8648c1444dba54d974cfa1a769: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:32:45.366027 5069 ts_tablet_manager.cc:1918] T 72c66b98ff0249268a5aa0a60e70851d P bc6ec79640c745879d85774d57594ab4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:32:45.366178 5069 log.cc:1196] T 72c66b98ff0249268a5aa0a60e70851d P bc6ec79640c745879d85774d57594ab4: Deleting WAL directory at /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-1/wal/wals/72c66b98ff0249268a5aa0a60e70851d
I20260926 07:32:45.366533 5068 ts_tablet_manager.cc:1918] T 72c66b98ff0249268a5aa0a60e70851d P 2f290d8648c1444dba54d974cfa1a769: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:32:45.366567 5069 ts_tablet_manager.cc:1939] T 72c66b98ff0249268a5aa0a60e70851d P bc6ec79640c745879d85774d57594ab4: Deleting consensus metadata
I20260926 07:32:45.366657 5068 log.cc:1196] T 72c66b98ff0249268a5aa0a60e70851d P 2f290d8648c1444dba54d974cfa1a769: Deleting WAL directory at /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-2/wal/wals/72c66b98ff0249268a5aa0a60e70851d
I20260926 07:32:45.366994 5068 ts_tablet_manager.cc:1939] T 72c66b98ff0249268a5aa0a60e70851d P 2f290d8648c1444dba54d974cfa1a769: Deleting consensus metadata
I20260926 07:32:45.368112 4564 catalog_manager.cc:4841] TS bc6ec79640c745879d85774d57594ab4 (127.22.53.194:43777): tablet 72c66b98ff0249268a5aa0a60e70851d (table test-table [id=d144eadd58a448fabc3973b516392202]) successfully deleted
I20260926 07:32:45.368186 4562 catalog_manager.cc:4841] TS 2f290d8648c1444dba54d974cfa1a769 (127.22.53.195:38919): tablet 72c66b98ff0249268a5aa0a60e70851d (table test-table [id=d144eadd58a448fabc3973b516392202]) successfully deleted
W20260926 07:32:45.369083 4563 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.53.193:37097 (error 108)
I20260926 07:32:45.369421 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 4770
I20260926 07:32:45.370927 5046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:45.370473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.193:37097 (local address 127.0.0.1:39580)
0926 07:32:45.370715 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:45.370800 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.193:37097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132}
W20260926 07:32:45.371132 4563 catalog_manager.cc:4584] TS 84670aa208b740d1bdde3276c1a33264 (127.22.53.193:37097): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 72c66b98ff0249268a5aa0a60e70851d: Network error: Client connection negotiation failed: client connection to 127.22.53.193:37097: connect: Connection refused (error 111)
I20260926 07:32:45.376714 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 4907
I20260926 07:32:45.383061 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 4545
I20260926 07:32:45.390571 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.53.254:35543
--webserver_interface=127.22.53.254
--webserver_port=37215
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab
--principal=kudu/127.22.53.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:46315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.53.254:35543
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:32:45.513311 5071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:32:45.513722 5071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:32:45.513808 5071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:32:45.518365 5071 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:32:45.518496 5071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:32:45.518532 5071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:32:45.518580 5071 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:32:45.518642 5071 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:32:45.524595 5071 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:46315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.53.254:35543
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab
--principal=kudu/127.22.53.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.53.254:35543
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.53.254
--webserver_port=37215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5071
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:32:45.525895 5071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:32:45.527073 5071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:32:45.536489 5077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:45.536504 5079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:32:45.536777 5071 server_base.cc:1034] running on GCE node
W20260926 07:32:45.536573 5076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:32:45.537545 5071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:32:45.538970 5071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:32:45.540211 5071 hybrid_clock.cc:648] HybridClock initialized: now 1790407965540179 us; error 54 us; skew 500 ppm
Sep 26 07:32:45 dist-test-slave-wdjc krb5kdc[3997](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407965, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:32:45.543440 5071 init.cc:377] Logged in from keytab as kudu/127.22.53.254@KRBTEST.COM (short username kudu)
I20260926 07:32:45.544467 5071 webserver.cc:460] Webserver started at http://127.22.53.254:37215/ using document root <none> and password file <none>
I20260926 07:32:45.545258 5071 fs_manager.cc:359] Metadata directory not provided
I20260926 07:32:45.545346 5071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:32:45.550040 5071 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260926 07:32:45.553436 5086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:45.555153 5071 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260926 07:32:45.555349 5071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/master-0/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/master-0/wal
uuid: "095dfa56bdf344f6b50d5cc18be740cd"
format_stamp: "Formatted at 2026-09-26 07:32:43 on dist-test-slave-wdjc"
server_key: "fa15f9a965a45d0401f978b68d44e8d6"
server_key_iv: "85062671625e999fae5efde875b2c3d0"
server_key_version: "encryptionkey@0"
I20260926 07:32:45.556018 5071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:32:45.572842 5071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:32:45.577790 5071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:32:45.578045 5071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:32:45.586308 5071 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.254:35543
I20260926 07:32:45.586650 5138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.254:35543 every 8 connection(s)
I20260926 07:32:45.588572 5071 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
I20260926 07:32:45.589365 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 5071
I20260926 07:32:45.589936 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.53.193:37097
--local_ip_for_outbound_sockets=127.22.53.193
--tserver_master_addrs=127.22.53.254:35543
--webserver_port=46039
--webserver_interface=127.22.53.193
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.193.keytab
--principal=kudu/127.22.53.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:46315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:32:45.594985 5139 sys_catalog.cc:263] Verifying existing consensus state
I20260926 07:32:45.597162 5139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 095dfa56bdf344f6b50d5cc18be740cd: Bootstrap starting.
I20260926 07:32:45.616313 5139 log.cc:824] T 00000000000000000000000000000000 P 095dfa56bdf344f6b50d5cc18be740cd: Log is configured to *not* fsync() on all Append() calls
I20260926 07:32:45.628686 5139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 095dfa56bdf344f6b50d5cc18be740cd: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 07:32:45.629148 5139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 095dfa56bdf344f6b50d5cc18be740cd: Bootstrap complete.
I20260926 07:32:45.633807 5139 raft_consensus.cc:348] T 00000000000000000000000000000000 P 095dfa56bdf344f6b50d5cc18be740cd [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "095dfa56bdf344f6b50d5cc18be740cd" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 35543 } }
I20260926 07:32:45.634351 5139 raft_consensus.cc:729] T 00000000000000000000000000000000 P 095dfa56bdf344f6b50d5cc18be740cd [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 095dfa56bdf344f6b50d5cc18be740cd, State: Initialized, Role: FOLLOWER
I20260926 07:32:45.634950 5139 consensus_queue.cc:260] T 00000000000000000000000000000000 P 095dfa56bdf344f6b50d5cc18be740cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "095dfa56bdf344f6b50d5cc18be740cd" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 35543 } }
I20260926 07:32:45.635154 5139 raft_consensus.cc:388] T 00000000000000000000000000000000 P 095dfa56bdf344f6b50d5cc18be740cd [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:32:45.635252 5139 raft_consensus.cc:482] T 00000000000000000000000000000000 P 095dfa56bdf344f6b50d5cc18be740cd [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:32:45.635385 5139 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 095dfa56bdf344f6b50d5cc18be740cd [term 2 FOLLOWER]: Advancing to term 3
I20260926 07:32:45.637723 5139 raft_consensus.cc:504] T 00000000000000000000000000000000 P 095dfa56bdf344f6b50d5cc18be740cd [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "095dfa56bdf344f6b50d5cc18be740cd" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 35543 } }
I20260926 07:32:45.638160 5139 leader_election.cc:302] T 00000000000000000000000000000000 P 095dfa56bdf344f6b50d5cc18be740cd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 095dfa56bdf344f6b50d5cc18be740cd; no voters:
I20260926 07:32:45.638551 5139 leader_election.cc:288] T 00000000000000000000000000000000 P 095dfa56bdf344f6b50d5cc18be740cd [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 07:32:45.638665 5143 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 095dfa56bdf344f6b50d5cc18be740cd [term 3 FOLLOWER]: Leader election won for term 3
I20260926 07:32:45.638890 5143 raft_consensus.cc:686] T 00000000000000000000000000000000 P 095dfa56bdf344f6b50d5cc18be740cd [term 3 LEADER]: Becoming Leader. State: Replica: 095dfa56bdf344f6b50d5cc18be740cd, State: Running, Role: LEADER
I20260926 07:32:45.639222 5143 consensus_queue.cc:237] T 00000000000000000000000000000000 P 095dfa56bdf344f6b50d5cc18be740cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "095dfa56bdf344f6b50d5cc18be740cd" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 35543 } }
I20260926 07:32:45.639961 5139 sys_catalog.cc:564] T 00000000000000000000000000000000 P 095dfa56bdf344f6b50d5cc18be740cd [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:32:45.640388 5145 sys_catalog.cc:455] T 00000000000000000000000000000000 P 095dfa56bdf344f6b50d5cc18be740cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 095dfa56bdf344f6b50d5cc18be740cd. Latest consensus state: current_term: 3 leader_uuid: "095dfa56bdf344f6b50d5cc18be740cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "095dfa56bdf344f6b50d5cc18be740cd" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 35543 } } }
I20260926 07:32:45.640561 5145 sys_catalog.cc:458] T 00000000000000000000000000000000 P 095dfa56bdf344f6b50d5cc18be740cd [sys.catalog]: This master's current role is: LEADER
I20260926 07:32:45.641011 5144 sys_catalog.cc:455] T 00000000000000000000000000000000 P 095dfa56bdf344f6b50d5cc18be740cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "095dfa56bdf344f6b50d5cc18be740cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "095dfa56bdf344f6b50d5cc18be740cd" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 35543 } } }
I20260926 07:32:45.641127 5144 sys_catalog.cc:458] T 00000000000000000000000000000000 P 095dfa56bdf344f6b50d5cc18be740cd [sys.catalog]: This master's current role is: LEADER
I20260926 07:32:45.642436 5147 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:32:45.648888 5147 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:32:45.649649 5147 catalog_manager.cc:1259] Loaded cluster ID: 96350a859a0d476b9eb3fc198c396466
I20260926 07:32:45.649742 5147 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260926 07:32:45.650697 5147 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fd91e74e9ad google::LogMessage::Fail() at ??:0
@ 0x7fd91e75188a google::LogMessage::SendToLog() at ??:0
@ 0x7fd91e74e4de google::LogMessage::Flush() at ??:0
@ 0x7fd91e74fa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fd92908dfdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fd92908ed2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fd92908b579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fd9290cf640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fd9209a03d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fd9209a0c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fd9209a23f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fd920991512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fd921fd56db start_thread at ??:0
@ 0x7fd91d24c61f clone at ??:0
W20260926 07:32:45.758972 5141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:32:45.759402 5141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:32:45.759474 5141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:32:45.766335 5141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:32:45.766469 5141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:32:45.766577 5141 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.53.193
I20260926 07:32:45.775841 5141 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:46315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.193.keytab
--principal=kudu/127.22.53.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.53.193:37097
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.53.193
--webserver_port=46039
--tserver_master_addrs=127.22.53.254:35543
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5141
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.53.193
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:32:45.777469 5141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:32:45.778966 5141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:32:45.791868 5180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:45.791855 5183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:45.791855 5181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:32:45.793526 5141 server_base.cc:1034] running on GCE node
I20260926 07:32:45.794112 5141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:32:45.794886 5141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:32:45.796104 5141 hybrid_clock.cc:648] HybridClock initialized: now 1790407965796082 us; error 63 us; skew 500 ppm
Sep 26 07:32:45 dist-test-slave-wdjc krb5kdc[3997](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407965, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:32:45.800364 5141 init.cc:377] Logged in from keytab as kudu/127.22.53.193@KRBTEST.COM (short username kudu)
I20260926 07:32:45.801924 5141 webserver.cc:460] Webserver started at http://127.22.53.193:46039/ using document root <none> and password file <none>
I20260926 07:32:45.802834 5141 fs_manager.cc:359] Metadata directory not provided
I20260926 07:32:45.802969 5141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:32:45.809170 5141 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 07:32:45.812949 5190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:45.814595 5141 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260926 07:32:45.814805 5141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-0/wal
uuid: "84670aa208b740d1bdde3276c1a33264"
format_stamp: "Formatted at 2026-09-26 07:32:43 on dist-test-slave-wdjc"
server_key: "5033f0687a2c951f94f1887fff755d26"
server_key_iv: "9c598874f2d5cacc1ad23cfdada1bf43"
server_key_version: "encryptionkey@0"
I20260926 07:32:45.815507 5141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:32:45.846413 5141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:32:45.850522 5141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:32:45.850802 5141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:32:45.851492 5141 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:32:45.853586 5197 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 07:32:45.860193 5141 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 07:32:45.860355 5141 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.001s sys 0.000s
I20260926 07:32:45.860455 5141 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 07:32:45.862522 5141 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 07:32:45.862624 5141 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260926 07:32:45.862979 5197 tablet_bootstrap.cc:492] T 72c66b98ff0249268a5aa0a60e70851d P 84670aa208b740d1bdde3276c1a33264: Bootstrap starting.
I20260926 07:32:45.874717 5141 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.193:37097
I20260926 07:32:45.875078 5304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.193:37097 every 8 connection(s)
I20260926 07:32:45.876235 5141 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-0/data/info.pb
I20260926 07:32:45.881402 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 5141
I20260926 07:32:45.881927 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.53.194:43777
--local_ip_for_outbound_sockets=127.22.53.194
--tserver_master_addrs=127.22.53.254:35543
--webserver_port=43399
--webserver_interface=127.22.53.194
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.194.keytab
--principal=kudu/127.22.53.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:46315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:32:45.885686 5197 log.cc:824] T 72c66b98ff0249268a5aa0a60e70851d P 84670aa208b740d1bdde3276c1a33264: Log is configured to *not* fsync() on all Append() calls
I20260926 07:32:45.893766 5197 tablet_bootstrap.cc:492] T 72c66b98ff0249268a5aa0a60e70851d P 84670aa208b740d1bdde3276c1a33264: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 07:32:45.894217 5197 tablet_bootstrap.cc:492] T 72c66b98ff0249268a5aa0a60e70851d P 84670aa208b740d1bdde3276c1a33264: Bootstrap complete.
I20260926 07:32:45.895388 5197 ts_tablet_manager.cc:1397] T 72c66b98ff0249268a5aa0a60e70851d P 84670aa208b740d1bdde3276c1a33264: Time spent bootstrapping tablet: real 0.033s user 0.021s sys 0.009s
I20260926 07:32:45.898083 5197 raft_consensus.cc:348] T 72c66b98ff0249268a5aa0a60e70851d P 84670aa208b740d1bdde3276c1a33264 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f290d8648c1444dba54d974cfa1a769" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 38919 } } peers { permanent_uuid: "bc6ec79640c745879d85774d57594ab4" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 43777 } } peers { permanent_uuid: "84670aa208b740d1bdde3276c1a33264" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 37097 } }
I20260926 07:32:45.898578 5197 raft_consensus.cc:729] T 72c66b98ff0249268a5aa0a60e70851d P 84670aa208b740d1bdde3276c1a33264 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84670aa208b740d1bdde3276c1a33264, State: Initialized, Role: FOLLOWER
I20260926 07:32:45.899115 5197 consensus_queue.cc:260] T 72c66b98ff0249268a5aa0a60e70851d P 84670aa208b740d1bdde3276c1a33264 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f290d8648c1444dba54d974cfa1a769" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 38919 } } peers { permanent_uuid: "bc6ec79640c745879d85774d57594ab4" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 43777 } } peers { permanent_uuid: "84670aa208b740d1bdde3276c1a33264" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 37097 } }
I20260926 07:32:45.900202 5197 ts_tablet_manager.cc:1428] T 72c66b98ff0249268a5aa0a60e70851d P 84670aa208b740d1bdde3276c1a33264: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
W20260926 07:32:45.928795 5307 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:32:45.878913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:35543 (local address 127.22.53.193:53271)
0926 07:32:45.879632 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:45.879704 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0926 07:32:45.881106 (+ 1402us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:32:45.927861 (+ 46755us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:35543: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":183,"threads_started":1}
W20260926 07:32:45.929492 5305 heartbeater.cc:644] Failed to heartbeat to 127.22.53.254:35543 (0 consecutive failures): Network error: Failed to ping master at 127.22.53.254:35543: Client connection negotiation failed: client connection to 127.22.53.254:35543: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 07:32:45.930423 5307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:45.930114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:35543 (local address 127.22.53.193:59173)
0926 07:32:45.930263 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:45.930354 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:35543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260926 07:32:45.931313 5307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:45.931008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:35543 (local address 127.22.53.193:41249)
0926 07:32:45.931174 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:45.931238 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:35543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260926 07:32:45.931541 5305 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 07:32:46.022643 5308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:32:46.022969 5308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:32:46.023018 5308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:32:46.027676 5308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:32:46.027774 5308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:32:46.027841 5308 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.53.194
I20260926 07:32:46.034017 5308 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:46315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.194.keytab
--principal=kudu/127.22.53.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.53.194:43777
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.53.194
--webserver_port=43399
--tserver_master_addrs=127.22.53.254:35543
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5308
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.53.194
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:32:46.035264 5308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:32:46.036396 5308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:32:46.047510 5316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:46.047518 5315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:46.047493 5318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:32:46.047822 5308 server_base.cc:1034] running on GCE node
I20260926 07:32:46.048404 5308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:32:46.049273 5308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:32:46.050513 5308 hybrid_clock.cc:648] HybridClock initialized: now 1790407966050489 us; error 38 us; skew 500 ppm
Sep 26 07:32:46 dist-test-slave-wdjc krb5kdc[3997](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407966, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:32:46.053988 5308 init.cc:377] Logged in from keytab as kudu/127.22.53.194@KRBTEST.COM (short username kudu)
I20260926 07:32:46.055114 5308 webserver.cc:460] Webserver started at http://127.22.53.194:43399/ using document root <none> and password file <none>
I20260926 07:32:46.055773 5308 fs_manager.cc:359] Metadata directory not provided
I20260926 07:32:46.055876 5308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:32:46.060525 5308 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:32:46.063689 5325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:46.065836 5308 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 07:32:46.066069 5308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-1/wal
uuid: "bc6ec79640c745879d85774d57594ab4"
format_stamp: "Formatted at 2026-09-26 07:32:43 on dist-test-slave-wdjc"
server_key: "ca7f2bc93ac94e8a8f22a2e9860eb593"
server_key_iv: "b22cde5e5514a23f3e64d49aef11e2fe"
server_key_version: "encryptionkey@0"
I20260926 07:32:46.066797 5308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:32:46.082172 5308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:32:46.086782 5308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:32:46.087085 5308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:32:46.087807 5308 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:32:46.088951 5308 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:32:46.089021 5308 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:46.089089 5308 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:32:46.089124 5308 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:46.102016 5308 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.194:43777
I20260926 07:32:46.102242 5438 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.194:43777 every 8 connection(s)
I20260926 07:32:46.103406 5308 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-1/data/info.pb
I20260926 07:32:46.109802 5441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:46.106911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:35543 (local address 127.22.53.194:51535)
0926 07:32:46.107617 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:46.107797 (+ 180us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:35543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":166,"threads_started":1}
W20260926 07:32:46.110702 5439 heartbeater.cc:644] Failed to heartbeat to 127.22.53.254:35543 (0 consecutive failures): Network error: Failed to ping master at 127.22.53.254:35543: Client connection negotiation failed: client connection to 127.22.53.254:35543: connect: Connection refused (error 111)
I20260926 07:32:46.111691 5441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:46.111314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:35543 (local address 127.22.53.194:42115)
0926 07:32:46.111528 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:46.111600 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:35543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260926 07:32:46.112668 5441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:46.112330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:35543 (local address 127.22.53.194:33551)
0926 07:32:46.112462 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:46.112521 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:35543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260926 07:32:46.112685 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 5308
W20260926 07:32:46.112875 5439 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 07:32:46.113340 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.53.195:38919
--local_ip_for_outbound_sockets=127.22.53.195
--tserver_master_addrs=127.22.53.254:35543
--webserver_port=41915
--webserver_interface=127.22.53.195
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.195.keytab
--principal=kudu/127.22.53.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:46315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:32:46.242586 5442 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:32:46.242974 5442 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:32:46.243077 5442 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:32:46.247561 5442 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:32:46.247710 5442 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:32:46.247812 5442 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.53.195
I20260926 07:32:46.253816 5442 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:46315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.195.keytab
--principal=kudu/127.22.53.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.53.195:38919
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.53.195
--webserver_port=41915
--tserver_master_addrs=127.22.53.254:35543
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5442
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.53.195
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:32:46.255057 5442 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:32:46.256276 5442 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:32:46.269068 5448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:46.269050 5447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:32:46.269040 5450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:32:46.269460 5442 server_base.cc:1034] running on GCE node
I20260926 07:32:46.270216 5442 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:32:46.271104 5442 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:32:46.272375 5442 hybrid_clock.cc:648] HybridClock initialized: now 1790407966272346 us; error 46 us; skew 500 ppm
Sep 26 07:32:46 dist-test-slave-wdjc krb5kdc[3997](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407966, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:32:46.276032 5442 init.cc:377] Logged in from keytab as kudu/127.22.53.195@KRBTEST.COM (short username kudu)
I20260926 07:32:46.277386 5442 webserver.cc:460] Webserver started at http://127.22.53.195:41915/ using document root <none> and password file <none>
I20260926 07:32:46.278105 5442 fs_manager.cc:359] Metadata directory not provided
I20260926 07:32:46.278234 5442 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:32:46.284219 5442 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 07:32:46.287024 5457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:46.289026 5442 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:32:46.289227 5442 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-2/wal
uuid: "2f290d8648c1444dba54d974cfa1a769"
format_stamp: "Formatted at 2026-09-26 07:32:43 on dist-test-slave-wdjc"
server_key: "481e228797e50ef9032d222ff42d3a8b"
server_key_iv: "3405b1c091992969b8ffbec0bbba738b"
server_key_version: "encryptionkey@0"
I20260926 07:32:46.290118 5442 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:32:46.308471 5442 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:32:46.313614 5442 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:32:46.313884 5442 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:32:46.314587 5442 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:32:46.315574 5442 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:32:46.315660 5442 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:46.315740 5442 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:32:46.315795 5442 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:32:46.327200 5442 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.195:38919
I20260926 07:32:46.327247 5570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.195:38919 every 8 connection(s)
I20260926 07:32:46.328595 5442 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407783529717-22743-0/minicluster-data/ts-2/data/info.pb
I20260926 07:32:46.333218 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 5442
I20260926 07:32:46.334767 5573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:46.331319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:35543 (local address 127.22.53.195:57967)
0926 07:32:46.332728 (+ 1409us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:46.333044 (+ 316us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:35543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":957,"thread_start_us":645,"threads_started":1}
W20260926 07:32:46.335443 5571 heartbeater.cc:644] Failed to heartbeat to 127.22.53.254:35543 (0 consecutive failures): Network error: Failed to ping master at 127.22.53.254:35543: Client connection negotiation failed: client connection to 127.22.53.254:35543: connect: Connection refused (error 111)
I20260926 07:32:46.336697 5573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:46.336161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:35543 (local address 127.22.53.195:44343)
0926 07:32:46.336392 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:46.336486 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:35543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260926 07:32:46.337893 5573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:46.337537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:35543 (local address 127.22.53.195:60021)
0926 07:32:46.337732 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:46.337798 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:35543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
W20260926 07:32:46.338202 5571 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 07:32:46.932991 5575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:46.932426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:35543 (local address 127.22.53.193:35111)
0926 07:32:46.932736 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:46.932818 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:35543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":121,"threads_started":1}
I20260926 07:32:47.114254 5576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:47.113680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:35543 (local address 127.22.53.194:41221)
0926 07:32:47.113995 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:47.114135 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:35543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":138,"threads_started":1}
I20260926 07:32:47.135052 5577 raft_consensus.cc:482] T 72c66b98ff0249268a5aa0a60e70851d P 84670aa208b740d1bdde3276c1a33264 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:32:47.135303 5577 raft_consensus.cc:504] T 72c66b98ff0249268a5aa0a60e70851d P 84670aa208b740d1bdde3276c1a33264 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f290d8648c1444dba54d974cfa1a769" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 38919 } } peers { permanent_uuid: "bc6ec79640c745879d85774d57594ab4" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 43777 } } peers { permanent_uuid: "84670aa208b740d1bdde3276c1a33264" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 37097 } }
I20260926 07:32:47.136775 5577 leader_election.cc:288] T 72c66b98ff0249268a5aa0a60e70851d P 84670aa208b740d1bdde3276c1a33264 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2f290d8648c1444dba54d974cfa1a769 (127.22.53.195:38919), bc6ec79640c745879d85774d57594ab4 (127.22.53.194:43777)
Sep 26 07:32:47 dist-test-slave-wdjc krb5kdc[3997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407965, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.193@KRBTEST.COM for kudu/127.22.53.194@KRBTEST.COM
Sep 26 07:32:47 dist-test-slave-wdjc krb5kdc[3997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407965, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.193@KRBTEST.COM for kudu/127.22.53.195@KRBTEST.COM
I20260926 07:32:47.153868 5575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:47.136970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.195:38919 (local address 127.22.53.193:43161)
0926 07:32:47.137079 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:47.137097 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 07:32:47.137222 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:32:47.139586 (+ 2364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:32:47.139592 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:32:47.139987 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:32:47.140680 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:47.140702 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:47.142093 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:47.142096 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:47.143653 (+ 1557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:47.143668 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:47.144613 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:47.144617 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:47.144803 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:47.145546 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:32:47.145588 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:32:47.147854 (+ 2266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:32:47.150669 (+ 2815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:32:47.150675 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:32:47.150688 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:32:47.150977 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:32:47.151430 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:32:47.151432 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:32:47.151433 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:32:47.151563 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:32:47.152462 (+ 899us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:32:47.152470 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:32:47.152923 (+ 453us) client_negotiation.cc:769] Sending connection context
0926 07:32:47.153177 (+ 254us) client_negotiation.cc:240] Negotiation successful
0926 07:32:47.153656 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":388,"spinlock_wait_cycles":33536}
I20260926 07:32:47.153879 5580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:47.137346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.193:50475 (local address 127.22.53.194:43777)
0926 07:32:47.137646 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:47.137679 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:47.137708 (+ 29us) server_negotiation.cc:407] Connection header received
0926 07:32:47.138603 (+ 895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:47.138610 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:47.139102 (+ 492us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:47.139467 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:47.140886 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:47.141895 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:47.143849 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:47.144431 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:47.144666 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:47.148090 (+ 3424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:47.148097 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:47.148117 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:32:47.148156 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:47.150506 (+ 2350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:47.151396 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:47.151405 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:47.151412 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:47.151557 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:47.151881 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:47.151884 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:47.151886 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:47.152312 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:47.152451 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:47.153354 (+ 903us) server_negotiation.cc:299] Negotiation successful
0926 07:32:47.153680 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":92,"threads_started":1}
I20260926 07:32:47.153883 5579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:47.137195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.193:43161 (local address 127.22.53.195:38919)
0926 07:32:47.137631 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0926 07:32:47.137648 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 07:32:47.137694 (+ 46us) server_negotiation.cc:407] Connection header received
0926 07:32:47.138596 (+ 902us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:32:47.138606 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:32:47.139102 (+ 496us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:32:47.139467 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:32:47.140879 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:47.141894 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:47.143812 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:32:47.144428 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:32:47.144666 (+ 238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:47.148089 (+ 3423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:32:47.148097 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:32:47.148117 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:32:47.148156 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:32:47.150506 (+ 2350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:47.151137 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:47.151143 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:47.151147 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:47.151283 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:32:47.151666 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:32:47.151669 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:32:47.151671 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:32:47.152295 (+ 624us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:32:47.152451 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0926 07:32:47.153354 (+ 903us) server_negotiation.cc:299] Negotiation successful
0926 07:32:47.153680 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":92,"threads_started":1}
I20260926 07:32:47.154597 5578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:47.137057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.194:43777 (local address 127.22.53.193:50475)
0926 07:32:47.137463 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:47.137477 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:32:47.137596 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:32:47.139979 (+ 2383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:32:47.139982 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:32:47.140147 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:32:47.140680 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:47.140702 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:47.142080 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:47.142084 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:47.143669 (+ 1585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:32:47.143685 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:32:47.144608 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:32:47.144612 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:32:47.144807 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:32:47.145582 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:32:47.145599 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:32:47.147369 (+ 1770us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:32:47.151096 (+ 3727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:32:47.151098 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:32:47.151099 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:32:47.151263 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:32:47.151703 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:32:47.151705 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:32:47.151708 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:32:47.151780 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:32:47.152462 (+ 682us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:32:47.152471 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:32:47.152923 (+ 452us) client_negotiation.cc:769] Sending connection context
0926 07:32:47.153177 (+ 254us) client_negotiation.cc:240] Negotiation successful
0926 07:32:47.154423 (+ 1246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"mutex_wait_us":1125,"spinlock_wait_cycles":9856,"thread_start_us":104,"threads_started":1}
W20260926 07:32:47.156992 5193 leader_election.cc:341] T 72c66b98ff0249268a5aa0a60e70851d P 84670aa208b740d1bdde3276c1a33264 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bc6ec79640c745879d85774d57594ab4 (127.22.53.194:43777): Not found: Tablet not found: 72c66b98ff0249268a5aa0a60e70851d
W20260926 07:32:47.157199 5191 leader_election.cc:341] T 72c66b98ff0249268a5aa0a60e70851d P 84670aa208b740d1bdde3276c1a33264 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2f290d8648c1444dba54d974cfa1a769 (127.22.53.195:38919): Not found: Tablet not found: 72c66b98ff0249268a5aa0a60e70851d
I20260926 07:32:47.157251 5193 leader_election.cc:302] T 72c66b98ff0249268a5aa0a60e70851d P 84670aa208b740d1bdde3276c1a33264 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 84670aa208b740d1bdde3276c1a33264; no voters: 2f290d8648c1444dba54d974cfa1a769, bc6ec79640c745879d85774d57594ab4
I20260926 07:32:47.157542 5577 raft_consensus.cc:2726] T 72c66b98ff0249268a5aa0a60e70851d P 84670aa208b740d1bdde3276c1a33264 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 07:32:47.339229 5581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:47.338829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:35543 (local address 127.22.53.195:43155)
0926 07:32:47.339067 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:47.339119 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:35543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":74,"threads_started":1}
I20260926 07:32:47.934543 5582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:47.934059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:35543 (local address 127.22.53.193:39541)
0926 07:32:47.934383 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:47.934444 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:35543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":137,"threads_started":1}
I20260926 07:32:48.115545 5583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:48.115081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:35543 (local address 127.22.53.194:39873)
0926 07:32:48.115394 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:48.115446 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:35543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":142,"threads_started":1}
I20260926 07:32:48.340507 5584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:48.340079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:35543 (local address 127.22.53.195:52847)
0926 07:32:48.340337 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:48.340393 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:35543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":77,"threads_started":1}
I20260926 07:32:48.697098 5585 raft_consensus.cc:482] T 72c66b98ff0249268a5aa0a60e70851d P 84670aa208b740d1bdde3276c1a33264 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:32:48.697239 5585 raft_consensus.cc:504] T 72c66b98ff0249268a5aa0a60e70851d P 84670aa208b740d1bdde3276c1a33264 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f290d8648c1444dba54d974cfa1a769" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 38919 } } peers { permanent_uuid: "bc6ec79640c745879d85774d57594ab4" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 43777 } } peers { permanent_uuid: "84670aa208b740d1bdde3276c1a33264" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 37097 } }
I20260926 07:32:48.697625 5585 leader_election.cc:288] T 72c66b98ff0249268a5aa0a60e70851d P 84670aa208b740d1bdde3276c1a33264 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2f290d8648c1444dba54d974cfa1a769 (127.22.53.195:38919), bc6ec79640c745879d85774d57594ab4 (127.22.53.194:43777)
W20260926 07:32:48.698523 5191 leader_election.cc:341] T 72c66b98ff0249268a5aa0a60e70851d P 84670aa208b740d1bdde3276c1a33264 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2f290d8648c1444dba54d974cfa1a769 (127.22.53.195:38919): Not found: Tablet not found: 72c66b98ff0249268a5aa0a60e70851d
W20260926 07:32:48.698623 5193 leader_election.cc:341] T 72c66b98ff0249268a5aa0a60e70851d P 84670aa208b740d1bdde3276c1a33264 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bc6ec79640c745879d85774d57594ab4 (127.22.53.194:43777): Not found: Tablet not found: 72c66b98ff0249268a5aa0a60e70851d
I20260926 07:32:48.698742 5193 leader_election.cc:302] T 72c66b98ff0249268a5aa0a60e70851d P 84670aa208b740d1bdde3276c1a33264 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 84670aa208b740d1bdde3276c1a33264; no voters: 2f290d8648c1444dba54d974cfa1a769, bc6ec79640c745879d85774d57594ab4
I20260926 07:32:48.698951 5585 raft_consensus.cc:2726] T 72c66b98ff0249268a5aa0a60e70851d P 84670aa208b740d1bdde3276c1a33264 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 07:32:48.935982 5586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:48.935453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:35543 (local address 127.22.53.193:49305)
0926 07:32:48.935816 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:48.935880 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:35543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":80,"threads_started":1}
I20260926 07:32:49.117079 5587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:49.116508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:35543 (local address 127.22.53.194:44619)
0926 07:32:49.116900 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:49.116967 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:35543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":96,"threads_started":1}
I20260926 07:32:49.341912 5588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:49.341449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:35543 (local address 127.22.53.195:60399)
0926 07:32:49.341724 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:49.341807 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:35543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":82,"threads_started":1}
I20260926 07:32:49.937472 5589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:49.936972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:35543 (local address 127.22.53.193:34219)
0926 07:32:49.937289 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:49.937358 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:35543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":84,"threads_started":1}
I20260926 07:32:50.118641 5590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:50.118086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:35543 (local address 127.22.53.194:57763)
0926 07:32:50.118414 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:50.118508 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:35543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":94,"threads_started":1}
I20260926 07:32:50.343436 5591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:50.342926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:35543 (local address 127.22.53.195:53811)
0926 07:32:50.343232 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:50.343332 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:35543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":79,"threads_started":1}
I20260926 07:32:50.939253 5592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:50.938760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:35543 (local address 127.22.53.193:59847)
0926 07:32:50.939083 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:50.939155 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:35543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":79,"threads_started":1}
I20260926 07:32:51.120237 5593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:51.119778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:35543 (local address 127.22.53.194:60803)
0926 07:32:51.120073 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:51.120138 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:35543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":93,"threads_started":1}
I20260926 07:32:51.247467 5594 raft_consensus.cc:482] T 72c66b98ff0249268a5aa0a60e70851d P 84670aa208b740d1bdde3276c1a33264 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:32:51.247615 5594 raft_consensus.cc:504] T 72c66b98ff0249268a5aa0a60e70851d P 84670aa208b740d1bdde3276c1a33264 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f290d8648c1444dba54d974cfa1a769" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 38919 } } peers { permanent_uuid: "bc6ec79640c745879d85774d57594ab4" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 43777 } } peers { permanent_uuid: "84670aa208b740d1bdde3276c1a33264" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 37097 } }
I20260926 07:32:51.248258 5594 leader_election.cc:288] T 72c66b98ff0249268a5aa0a60e70851d P 84670aa208b740d1bdde3276c1a33264 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2f290d8648c1444dba54d974cfa1a769 (127.22.53.195:38919), bc6ec79640c745879d85774d57594ab4 (127.22.53.194:43777)
W20260926 07:32:51.249290 5193 leader_election.cc:341] T 72c66b98ff0249268a5aa0a60e70851d P 84670aa208b740d1bdde3276c1a33264 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bc6ec79640c745879d85774d57594ab4 (127.22.53.194:43777): Not found: Tablet not found: 72c66b98ff0249268a5aa0a60e70851d
W20260926 07:32:51.249425 5191 leader_election.cc:341] T 72c66b98ff0249268a5aa0a60e70851d P 84670aa208b740d1bdde3276c1a33264 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2f290d8648c1444dba54d974cfa1a769 (127.22.53.195:38919): Not found: Tablet not found: 72c66b98ff0249268a5aa0a60e70851d
I20260926 07:32:51.249480 5191 leader_election.cc:302] T 72c66b98ff0249268a5aa0a60e70851d P 84670aa208b740d1bdde3276c1a33264 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 84670aa208b740d1bdde3276c1a33264; no voters: 2f290d8648c1444dba54d974cfa1a769, bc6ec79640c745879d85774d57594ab4
I20260926 07:32:51.249653 5594 raft_consensus.cc:2726] T 72c66b98ff0249268a5aa0a60e70851d P 84670aa208b740d1bdde3276c1a33264 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 07:32:51.345049 5595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:51.344517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:35543 (local address 127.22.53.195:35517)
0926 07:32:51.344873 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:51.344943 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:35543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":83,"threads_started":1}
I20260926 07:32:51.941022 5596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:51.940489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:35543 (local address 127.22.53.193:45647)
0926 07:32:51.940856 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:51.940922 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:35543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":148,"threads_started":1}
I20260926 07:32:52.121542 5597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:52.121070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:35543 (local address 127.22.53.194:44865)
0926 07:32:52.121373 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:52.121437 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:35543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":82,"threads_started":1}
I20260926 07:32:52.346395 5598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:52.345876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:35543 (local address 127.22.53.195:36971)
0926 07:32:52.346222 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:52.346290 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:35543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":99,"threads_started":1}
I20260926 07:32:52.942634 5599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:52.942061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:35543 (local address 127.22.53.193:59415)
0926 07:32:52.942425 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:52.942528 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:35543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":93,"threads_started":1}
I20260926 07:32:53.123898 5600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:53.123328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:35543 (local address 127.22.53.194:57429)
0926 07:32:53.123732 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:53.123796 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:35543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":87,"threads_started":1}
I20260926 07:32:53.347899 5601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:53.347352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:35543 (local address 127.22.53.195:35087)
0926 07:32:53.347732 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:53.347799 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:35543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":93,"threads_started":1}
I20260926 07:32:53.944494 5602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:53.943859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:35543 (local address 127.22.53.193:59239)
0926 07:32:53.944295 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:53.944380 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:35543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":122,"threads_started":1}
I20260926 07:32:54.125794 5603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:54.125270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:35543 (local address 127.22.53.194:33805)
0926 07:32:54.125550 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:54.125620 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:35543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":97,"threads_started":1}
I20260926 07:32:54.211068 5604 raft_consensus.cc:482] T 72c66b98ff0249268a5aa0a60e70851d P 84670aa208b740d1bdde3276c1a33264 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:32:54.211299 5604 raft_consensus.cc:504] T 72c66b98ff0249268a5aa0a60e70851d P 84670aa208b740d1bdde3276c1a33264 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f290d8648c1444dba54d974cfa1a769" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 38919 } } peers { permanent_uuid: "bc6ec79640c745879d85774d57594ab4" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 43777 } } peers { permanent_uuid: "84670aa208b740d1bdde3276c1a33264" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 37097 } }
I20260926 07:32:54.211849 5604 leader_election.cc:288] T 72c66b98ff0249268a5aa0a60e70851d P 84670aa208b740d1bdde3276c1a33264 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2f290d8648c1444dba54d974cfa1a769 (127.22.53.195:38919), bc6ec79640c745879d85774d57594ab4 (127.22.53.194:43777)
W20260926 07:32:54.212836 5191 leader_election.cc:341] T 72c66b98ff0249268a5aa0a60e70851d P 84670aa208b740d1bdde3276c1a33264 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2f290d8648c1444dba54d974cfa1a769 (127.22.53.195:38919): Not found: Tablet not found: 72c66b98ff0249268a5aa0a60e70851d
W20260926 07:32:54.212994 5193 leader_election.cc:341] T 72c66b98ff0249268a5aa0a60e70851d P 84670aa208b740d1bdde3276c1a33264 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bc6ec79640c745879d85774d57594ab4 (127.22.53.194:43777): Not found: Tablet not found: 72c66b98ff0249268a5aa0a60e70851d
I20260926 07:32:54.213088 5193 leader_election.cc:302] T 72c66b98ff0249268a5aa0a60e70851d P 84670aa208b740d1bdde3276c1a33264 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 84670aa208b740d1bdde3276c1a33264; no voters: 2f290d8648c1444dba54d974cfa1a769, bc6ec79640c745879d85774d57594ab4
I20260926 07:32:54.213259 5604 raft_consensus.cc:2726] T 72c66b98ff0249268a5aa0a60e70851d P 84670aa208b740d1bdde3276c1a33264 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 07:32:54.349876 5605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:54.349378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:35543 (local address 127.22.53.195:47175)
0926 07:32:54.349694 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:54.349762 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:35543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":96,"threads_started":1}
I20260926 07:32:54.946174 5606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:54.945596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:35543 (local address 127.22.53.193:42817)
0926 07:32:54.945924 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:54.946001 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:35543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":81,"threads_started":1}
I20260926 07:32:55.127842 5607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:55.127137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:35543 (local address 127.22.53.194:34651)
0926 07:32:55.127621 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:55.127710 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:35543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":146,"threads_started":1}
I20260926 07:32:55.351526 5608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:55.350927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:35543 (local address 127.22.53.195:49205)
0926 07:32:55.351351 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:55.351422 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:35543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":136,"threads_started":1}
I20260926 07:32:55.947911 5609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:55.947376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:35543 (local address 127.22.53.193:60115)
0926 07:32:55.947742 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:55.947808 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:35543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":84,"threads_started":1}
I20260926 07:32:56.129380 5610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:56.128903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:35543 (local address 127.22.53.194:43245)
0926 07:32:56.129211 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:56.129278 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:35543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":83,"threads_started":1}
I20260926 07:32:56.353036 5611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:56.352502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:35543 (local address 127.22.53.195:50797)
0926 07:32:56.352874 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:56.352936 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:35543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":156,"threads_started":1}
I20260926 07:32:56.949326 5612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:56.948870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:35543 (local address 127.22.53.193:51467)
0926 07:32:56.949137 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:56.949197 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:35543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":76,"threads_started":1}
I20260926 07:32:57.131194 5613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:57.130345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:35543 (local address 127.22.53.194:48113)
0926 07:32:57.130667 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:57.131073 (+ 406us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:35543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":85,"threads_started":1}
I20260926 07:32:57.243831 5614 raft_consensus.cc:482] T 72c66b98ff0249268a5aa0a60e70851d P 84670aa208b740d1bdde3276c1a33264 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:32:57.244023 5614 raft_consensus.cc:504] T 72c66b98ff0249268a5aa0a60e70851d P 84670aa208b740d1bdde3276c1a33264 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f290d8648c1444dba54d974cfa1a769" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 38919 } } peers { permanent_uuid: "bc6ec79640c745879d85774d57594ab4" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 43777 } } peers { permanent_uuid: "84670aa208b740d1bdde3276c1a33264" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 37097 } }
I20260926 07:32:57.244609 5614 leader_election.cc:288] T 72c66b98ff0249268a5aa0a60e70851d P 84670aa208b740d1bdde3276c1a33264 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2f290d8648c1444dba54d974cfa1a769 (127.22.53.195:38919), bc6ec79640c745879d85774d57594ab4 (127.22.53.194:43777)
W20260926 07:32:57.245481 5193 leader_election.cc:341] T 72c66b98ff0249268a5aa0a60e70851d P 84670aa208b740d1bdde3276c1a33264 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bc6ec79640c745879d85774d57594ab4 (127.22.53.194:43777): Not found: Tablet not found: 72c66b98ff0249268a5aa0a60e70851d
W20260926 07:32:57.245651 5191 leader_election.cc:341] T 72c66b98ff0249268a5aa0a60e70851d P 84670aa208b740d1bdde3276c1a33264 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2f290d8648c1444dba54d974cfa1a769 (127.22.53.195:38919): Not found: Tablet not found: 72c66b98ff0249268a5aa0a60e70851d
I20260926 07:32:57.245749 5191 leader_election.cc:302] T 72c66b98ff0249268a5aa0a60e70851d P 84670aa208b740d1bdde3276c1a33264 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 84670aa208b740d1bdde3276c1a33264; no voters: 2f290d8648c1444dba54d974cfa1a769, bc6ec79640c745879d85774d57594ab4
I20260926 07:32:57.245939 5614 raft_consensus.cc:2726] T 72c66b98ff0249268a5aa0a60e70851d P 84670aa208b740d1bdde3276c1a33264 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 07:32:57.354511 5615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:57.354052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:35543 (local address 127.22.53.195:33921)
0926 07:32:57.354356 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:57.354412 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:35543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":89,"threads_started":1}
I20260926 07:32:57.950953 5616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:57.950285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:35543 (local address 127.22.53.193:54307)
0926 07:32:57.950700 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:57.950801 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:35543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":102,"threads_started":1}
I20260926 07:32:58.132992 5617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:58.132430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:35543 (local address 127.22.53.194:34313)
0926 07:32:58.132812 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:58.132883 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:35543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":134,"threads_started":1}
I20260926 07:32:58.356328 5618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:58.355558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:35543 (local address 127.22.53.195:42283)
0926 07:32:58.356141 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:58.356223 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:35543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":116,"threads_started":1}
I20260926 07:32:58.952670 5619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:58.952102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:35543 (local address 127.22.53.193:58575)
0926 07:32:58.952402 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:58.952469 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:35543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":83,"threads_started":1}
I20260926 07:32:59.134725 5620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:59.134322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:35543 (local address 127.22.53.194:53031)
0926 07:32:59.134581 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:59.134630 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:35543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":84,"threads_started":1}
I20260926 07:32:59.358050 5621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:59.357510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:35543 (local address 127.22.53.195:54671)
0926 07:32:59.357857 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:59.357928 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:35543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":119,"threads_started":1}
I20260926 07:32:59.954195 5622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:32:59.953672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:35543 (local address 127.22.53.193:45499)
0926 07:32:59.953994 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0926 07:32:59.954081 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:35543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":82,"threads_started":1}
I20260926 07:33:00.135931 5623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:00.135487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:35543 (local address 127.22.53.194:60537)
0926 07:33:00.135762 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:00.135823 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:35543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":87,"threads_started":1}
I20260926 07:33:00.359476 5624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:00.358946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:35543 (local address 127.22.53.195:48303)
0926 07:33:00.359282 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:00.359348 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:35543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":99,"threads_started":1}
I20260926 07:33:00.956023 5625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:00.955483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:35543 (local address 127.22.53.193:44779)
0926 07:33:00.955813 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:00.955895 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:35543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":84,"threads_started":1}
I20260926 07:33:01.138103 5626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:01.137413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:35543 (local address 127.22.53.194:46563)
0926 07:33:01.137809 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:01.137889 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:35543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":92,"threads_started":1}
I20260926 07:33:01.342331 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 5141
I20260926 07:33:01.348717 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 5308
I20260926 07:33:01.354440 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 5442
2026-09-26T07:33:01Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19543 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:33:01 dist-test-slave-wdjc krb5kdc[5629](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:33:01 dist-test-slave-wdjc krb5kdc[5629](info): set up 2 sockets
Sep 26 07:33:01 dist-test-slave-wdjc krb5kdc[5629](info): commencing operation
krb5kdc: starting...
W20260926 07:33:02.441427 22743 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/krb5kdc/test-user.keytab.
Sep 26 07:33:02 dist-test-slave-wdjc krb5kdc[5629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407982, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:33:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:33:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.53.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.53.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.53.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab.
Entry for principal kudu/127.22.53.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab.
I20260926 07:33:02.584517 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.53.254:32891
--webserver_interface=127.22.53.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab
--principal=kudu/127.22.53.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:36403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.53.254:32891
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:33:02.706328 5642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:33:02.706673 5642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:33:02.706719 5642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:33:02.711316 5642 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 07:33:02.711404 5642 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 07:33:02.711422 5642 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 07:33:02.711437 5642 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:33:02.711458 5642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:33:02.711476 5642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:33:02.711493 5642 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:33:02.711510 5642 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:33:02.717658 5642 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:36403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.22.53.254:32891
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab
--principal=kudu/127.22.53.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.53.254:32891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.53.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5642
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:33:02.718865 5642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:33:02.720863 5642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:33:02.731753 5647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:02.731727 5648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:33:02.731794 5642 server_base.cc:1034] running on GCE node
W20260926 07:33:02.731727 5650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:33:02.733294 5642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:33:02.734911 5642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:33:02.736181 5642 hybrid_clock.cc:648] HybridClock initialized: now 1790407982736138 us; error 57 us; skew 500 ppm
Sep 26 07:33:02 dist-test-slave-wdjc krb5kdc[5629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407982, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:33:02.743296 5642 init.cc:377] Logged in from keytab as kudu/127.22.53.254@KRBTEST.COM (short username kudu)
I20260926 07:33:02.744477 5642 webserver.cc:460] Webserver started at http://127.22.53.254:37697/ using document root <none> and password file <none>
I20260926 07:33:02.745241 5642 fs_manager.cc:359] Metadata directory not provided
I20260926 07:33:02.745362 5642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:33:02.745641 5642 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:33:02.748044 5642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/master-0/data/instance:
uuid: "788e7a47032d4f458d5493fcf0021f9d"
format_stamp: "Formatted at 2026-09-26 07:33:02 on dist-test-slave-wdjc"
server_key: "299243d4b6966ad5336afd5e0b3334a4"
server_key_iv: "6e2a817fc2ea3438e7c194a94f3aabfa"
server_key_version: "encryptionkey@0"
I20260926 07:33:02.748850 5642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/master-0/wal/instance:
uuid: "788e7a47032d4f458d5493fcf0021f9d"
format_stamp: "Formatted at 2026-09-26 07:33:02 on dist-test-slave-wdjc"
server_key: "299243d4b6966ad5336afd5e0b3334a4"
server_key_iv: "6e2a817fc2ea3438e7c194a94f3aabfa"
server_key_version: "encryptionkey@0"
I20260926 07:33:02.753824 5642 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260926 07:33:02.757053 5657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:02.758731 5642 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 07:33:02.758944 5642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/master-0/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/master-0/wal
uuid: "788e7a47032d4f458d5493fcf0021f9d"
format_stamp: "Formatted at 2026-09-26 07:33:02 on dist-test-slave-wdjc"
server_key: "299243d4b6966ad5336afd5e0b3334a4"
server_key_iv: "6e2a817fc2ea3438e7c194a94f3aabfa"
server_key_version: "encryptionkey@0"
I20260926 07:33:02.759104 5642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:33:02.788200 5642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:33:02.792443 5642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:33:02.792795 5642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:33:02.801681 5642 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.254:32891
I20260926 07:33:02.802031 5709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.254:32891 every 8 connection(s)
I20260926 07:33:02.802973 5642 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
I20260926 07:33:02.803153 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 5642
I20260926 07:33:02.803280 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/master-0/wal/instance
I20260926 07:33:02.803674 22743 external_mini_cluster.cc:1428] Setting key 03b869fe9cbc40ff1940d77421191e8e
I20260926 07:33:02.806963 5710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 07:33:02 dist-test-slave-wdjc krb5kdc[5629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407982, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.53.254@KRBTEST.COM
I20260926 07:33:02.815045 5710 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 788e7a47032d4f458d5493fcf0021f9d: Bootstrap starting.
I20260926 07:33:02.818157 5710 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 788e7a47032d4f458d5493fcf0021f9d: Neither blocks nor log segments found. Creating new log.
I20260926 07:33:02.819130 5710 log.cc:824] T 00000000000000000000000000000000 P 788e7a47032d4f458d5493fcf0021f9d: Log is configured to *not* fsync() on all Append() calls
I20260926 07:33:02.821923 5710 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 788e7a47032d4f458d5493fcf0021f9d: No bootstrap required, opened a new log
I20260926 07:33:02.822139 5713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:02.805417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34498 (local address 127.22.53.254:32891)
0926 07:33:02.806036 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:02.806055 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:02.806132 (+ 77us) server_negotiation.cc:407] Connection header received
0926 07:33:02.806855 (+ 723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:02.806875 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:02.807364 (+ 489us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:02.807773 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:02.808756 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:02.809840 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:02.810795 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:02.811343 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:02.811550 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:02.815503 (+ 3953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:02.815509 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:02.815527 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:02.815577 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:02.818375 (+ 2798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:02.819025 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:02.819030 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:02.819036 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:02.819184 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:02.819528 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:02.819537 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:02.819538 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:02.820243 (+ 705us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:02.820503 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:02.820882 (+ 379us) server_negotiation.cc:299] Negotiation successful
0926 07:33:02.821241 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":197,"threads_started":1}
I20260926 07:33:02.826398 5710 raft_consensus.cc:348] T 00000000000000000000000000000000 P 788e7a47032d4f458d5493fcf0021f9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788e7a47032d4f458d5493fcf0021f9d" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 32891 } }
I20260926 07:33:02.826750 5710 raft_consensus.cc:374] T 00000000000000000000000000000000 P 788e7a47032d4f458d5493fcf0021f9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:33:02.826845 5710 raft_consensus.cc:729] T 00000000000000000000000000000000 P 788e7a47032d4f458d5493fcf0021f9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 788e7a47032d4f458d5493fcf0021f9d, State: Initialized, Role: FOLLOWER
I20260926 07:33:02.827342 5710 consensus_queue.cc:260] T 00000000000000000000000000000000 P 788e7a47032d4f458d5493fcf0021f9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788e7a47032d4f458d5493fcf0021f9d" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 32891 } }
I20260926 07:33:02.827515 5710 raft_consensus.cc:388] T 00000000000000000000000000000000 P 788e7a47032d4f458d5493fcf0021f9d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:33:02.827612 5710 raft_consensus.cc:482] T 00000000000000000000000000000000 P 788e7a47032d4f458d5493fcf0021f9d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:33:02.827761 5710 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 788e7a47032d4f458d5493fcf0021f9d [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:33:02.829082 5710 raft_consensus.cc:504] T 00000000000000000000000000000000 P 788e7a47032d4f458d5493fcf0021f9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788e7a47032d4f458d5493fcf0021f9d" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 32891 } }
I20260926 07:33:02.829547 5710 leader_election.cc:302] T 00000000000000000000000000000000 P 788e7a47032d4f458d5493fcf0021f9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 788e7a47032d4f458d5493fcf0021f9d; no voters:
I20260926 07:33:02.830034 5710 leader_election.cc:288] T 00000000000000000000000000000000 P 788e7a47032d4f458d5493fcf0021f9d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:33:02.830598 5715 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 788e7a47032d4f458d5493fcf0021f9d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:33:02.831208 5715 raft_consensus.cc:686] T 00000000000000000000000000000000 P 788e7a47032d4f458d5493fcf0021f9d [term 1 LEADER]: Becoming Leader. State: Replica: 788e7a47032d4f458d5493fcf0021f9d, State: Running, Role: LEADER
I20260926 07:33:02.831652 5715 consensus_queue.cc:237] T 00000000000000000000000000000000 P 788e7a47032d4f458d5493fcf0021f9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788e7a47032d4f458d5493fcf0021f9d" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 32891 } }
I20260926 07:33:02.831756 5710 sys_catalog.cc:564] T 00000000000000000000000000000000 P 788e7a47032d4f458d5493fcf0021f9d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:33:02.833509 5717 sys_catalog.cc:455] T 00000000000000000000000000000000 P 788e7a47032d4f458d5493fcf0021f9d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 788e7a47032d4f458d5493fcf0021f9d. Latest consensus state: current_term: 1 leader_uuid: "788e7a47032d4f458d5493fcf0021f9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788e7a47032d4f458d5493fcf0021f9d" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 32891 } } }
I20260926 07:33:02.833570 5716 sys_catalog.cc:455] T 00000000000000000000000000000000 P 788e7a47032d4f458d5493fcf0021f9d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "788e7a47032d4f458d5493fcf0021f9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788e7a47032d4f458d5493fcf0021f9d" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 32891 } } }
I20260926 07:33:02.833679 5717 sys_catalog.cc:458] T 00000000000000000000000000000000 P 788e7a47032d4f458d5493fcf0021f9d [sys.catalog]: This master's current role is: LEADER
I20260926 07:33:02.833745 5716 sys_catalog.cc:458] T 00000000000000000000000000000000 P 788e7a47032d4f458d5493fcf0021f9d [sys.catalog]: This master's current role is: LEADER
I20260926 07:33:02.834200 5727 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:33:02.838099 5727 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:33:02.846676 5727 catalog_manager.cc:1347] Generated new cluster ID: 8716aa67f32a4bd89be30c51639b80e4
I20260926 07:33:02.846788 5727 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:33:02.871263 5727 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:33:02.872838 5727 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:33:02.890642 5727 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 788e7a47032d4f458d5493fcf0021f9d: Generated new TSK 0
I20260926 07:33:02.891777 5727 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.53.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.53.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.53.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.193.keytab.
Entry for principal kudu/127.22.53.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.193.keytab.
I20260926 07:33:02.933109 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.53.193:0
--local_ip_for_outbound_sockets=127.22.53.193
--webserver_interface=127.22.53.193
--webserver_port=0
--tserver_master_addrs=127.22.53.254:32891
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.193.keytab
--principal=kudu/127.22.53.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:36403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:33:03.056634 5736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:33:03.056957 5736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:33:03.057005 5736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:33:03.061357 5736 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:33:03.061436 5736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:33:03.061501 5736 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.53.193
I20260926 07:33:03.069308 5736 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:36403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.193.keytab
--principal=kudu/127.22.53.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.53.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.53.193
--webserver_port=0
--tserver_master_addrs=127.22.53.254:32891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5736
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.53.193
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:33:03.070633 5736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:33:03.071838 5736 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:33:03.083199 5741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:03.083196 5742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:03.083199 5744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:33:03.083304 5736 server_base.cc:1034] running on GCE node
I20260926 07:33:03.083976 5736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:33:03.084880 5736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:33:03.086077 5736 hybrid_clock.cc:648] HybridClock initialized: now 1790407983086062 us; error 34 us; skew 500 ppm
Sep 26 07:33:03 dist-test-slave-wdjc krb5kdc[5629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407983, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:33:03.089434 5736 init.cc:377] Logged in from keytab as kudu/127.22.53.193@KRBTEST.COM (short username kudu)
I20260926 07:33:03.090644 5736 webserver.cc:460] Webserver started at http://127.22.53.193:46431/ using document root <none> and password file <none>
I20260926 07:33:03.091517 5736 fs_manager.cc:359] Metadata directory not provided
I20260926 07:33:03.091650 5736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:33:03.091910 5736 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:33:03.094408 5736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-0/data/instance:
uuid: "35ab6a8afa1a4639b35923c7d881a6c7"
format_stamp: "Formatted at 2026-09-26 07:33:03 on dist-test-slave-wdjc"
server_key: "e1ffe45e46c039841ee805ddd4d70b4d"
server_key_iv: "d0b5410a929a21d0fd39ce354f1d6709"
server_key_version: "encryptionkey@0"
I20260926 07:33:03.095082 5736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-0/wal/instance:
uuid: "35ab6a8afa1a4639b35923c7d881a6c7"
format_stamp: "Formatted at 2026-09-26 07:33:03 on dist-test-slave-wdjc"
server_key: "e1ffe45e46c039841ee805ddd4d70b4d"
server_key_iv: "d0b5410a929a21d0fd39ce354f1d6709"
server_key_version: "encryptionkey@0"
I20260926 07:33:03.099520 5736 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260926 07:33:03.102694 5751 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:03.104229 5736 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 07:33:03.104430 5736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-0/wal
uuid: "35ab6a8afa1a4639b35923c7d881a6c7"
format_stamp: "Formatted at 2026-09-26 07:33:03 on dist-test-slave-wdjc"
server_key: "e1ffe45e46c039841ee805ddd4d70b4d"
server_key_iv: "d0b5410a929a21d0fd39ce354f1d6709"
server_key_version: "encryptionkey@0"
I20260926 07:33:03.104689 5736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:33:03.118654 5736 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:33:03.122802 5736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:33:03.123234 5736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:33:03.124023 5736 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:33:03.125073 5736 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:33:03.125161 5736 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:03.125238 5736 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:33:03.125284 5736 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:03.135895 5736 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.193:40583
I20260926 07:33:03.135918 5864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.193:40583 every 8 connection(s)
I20260926 07:33:03.136981 5736 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-0/data/info.pb
I20260926 07:33:03.141759 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 5736
I20260926 07:33:03.141942 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-0/wal/instance
I20260926 07:33:03.142271 22743 external_mini_cluster.cc:1428] Setting key cbd5ce746cea13ae34c22ff7fefd2167
Sep 26 07:33:03 dist-test-slave-wdjc krb5kdc[5629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407983, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.193@KRBTEST.COM for kudu/127.22.53.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.22.53.194@KRBTEST.COM; defaulting to no policy
I20260926 07:33:03.153996 5713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:03.139058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.193:33419 (local address 127.22.53.254:32891)
0926 07:33:03.139211 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:03.139216 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:03.140113 (+ 897us) server_negotiation.cc:407] Connection header received
0926 07:33:03.141742 (+ 1629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:03.141745 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:03.141818 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:03.141958 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:03.143549 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:03.144242 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:03.146376 (+ 2134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:03.146639 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:03.146763 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:03.149743 (+ 2980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:03.149748 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:03.149750 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:03.149780 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:03.151491 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:03.152131 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:03.152135 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:03.152136 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:03.152215 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:03.152638 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:03.152642 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:03.152643 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:03.152883 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:03.153007 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:03.153646 (+ 639us) server_negotiation.cc:299] Negotiation successful
0926 07:33:03.153807 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.53.194@KRBTEST.COM" created.
I20260926 07:33:03.154979 5867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:03.139428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:32891 (local address 127.22.53.193:33419)
0926 07:33:03.139953 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:03.140001 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 07:33:03.141410 (+ 1409us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:33:03.142204 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:33:03.142216 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:33:03.142745 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:33:03.143379 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:03.143395 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:03.144410 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:03.144416 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:03.146209 (+ 1793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:03.146224 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:03.146822 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:03.146827 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:03.147010 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:03.147749 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:33:03.147768 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:33:03.149502 (+ 1734us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:33:03.151658 (+ 2156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:33:03.151665 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:33:03.151686 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:33:03.152003 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:33:03.152317 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:33:03.152321 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:33:03.152323 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:33:03.152487 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:33:03.153008 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:33:03.153016 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:33:03.153350 (+ 334us) client_negotiation.cc:769] Sending connection context
0926 07:33:03.153589 (+ 239us) client_negotiation.cc:240] Negotiation successful
0926 07:33:03.153972 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":139,"threads_started":1}
I20260926 07:33:03.156401 5865 heartbeater.cc:347] Connected to a master server at 127.22.53.254:32891
I20260926 07:33:03.156734 5865 heartbeater.cc:464] Registering TS with master...
I20260926 07:33:03.157342 5865 heartbeater.cc:511] Master 127.22.53.254:32891 requested a full tablet report, sending...
I20260926 07:33:03.159103 5674 ts_manager.cc:194] Registered new tserver with Master: 35ab6a8afa1a4639b35923c7d881a6c7 (127.22.53.193:40583)
I20260926 07:33:03.160367 5674 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.53.193@KRBTEST.COM'} at 127.22.53.193:33419
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.53.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.194.keytab.
Entry for principal kudu/127.22.53.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.194.keytab.
I20260926 07:33:03.176900 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.53.194:0
--local_ip_for_outbound_sockets=127.22.53.194
--webserver_interface=127.22.53.194
--webserver_port=0
--tserver_master_addrs=127.22.53.254:32891
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.194.keytab
--principal=kudu/127.22.53.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:36403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:33:03.302909 5870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:33:03.303249 5870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:33:03.303298 5870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:33:03.307969 5870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:33:03.308063 5870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:33:03.308131 5870 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.53.194
I20260926 07:33:03.314422 5870 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:36403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.194.keytab
--principal=kudu/127.22.53.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.53.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.53.194
--webserver_port=0
--tserver_master_addrs=127.22.53.254:32891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5870
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.53.194
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:33:03.315732 5870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:33:03.316908 5870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:33:03.328404 5875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:03.328404 5876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:33:03.328569 5870 server_base.cc:1034] running on GCE node
W20260926 07:33:03.328768 5878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:33:03.329200 5870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:33:03.329977 5870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:33:03.331205 5870 hybrid_clock.cc:648] HybridClock initialized: now 1790407983331188 us; error 34 us; skew 500 ppm
Sep 26 07:33:03 dist-test-slave-wdjc krb5kdc[5629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407983, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:33:03.334722 5870 init.cc:377] Logged in from keytab as kudu/127.22.53.194@KRBTEST.COM (short username kudu)
I20260926 07:33:03.335817 5870 webserver.cc:460] Webserver started at http://127.22.53.194:36643/ using document root <none> and password file <none>
I20260926 07:33:03.336489 5870 fs_manager.cc:359] Metadata directory not provided
I20260926 07:33:03.336623 5870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:33:03.336820 5870 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:33:03.339004 5870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-1/data/instance:
uuid: "ed471121813d4d1a9eb3f4e1c02da65b"
format_stamp: "Formatted at 2026-09-26 07:33:03 on dist-test-slave-wdjc"
server_key: "106f575b8ff401addcc90eff49306219"
server_key_iv: "ba0883ae30d30c700a88b832a8a47804"
server_key_version: "encryptionkey@0"
I20260926 07:33:03.339591 5870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-1/wal/instance:
uuid: "ed471121813d4d1a9eb3f4e1c02da65b"
format_stamp: "Formatted at 2026-09-26 07:33:03 on dist-test-slave-wdjc"
server_key: "106f575b8ff401addcc90eff49306219"
server_key_iv: "ba0883ae30d30c700a88b832a8a47804"
server_key_version: "encryptionkey@0"
I20260926 07:33:03.344288 5870 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260926 07:33:03.347381 5885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:03.348963 5870 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:33:03.349161 5870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-1/wal
uuid: "ed471121813d4d1a9eb3f4e1c02da65b"
format_stamp: "Formatted at 2026-09-26 07:33:03 on dist-test-slave-wdjc"
server_key: "106f575b8ff401addcc90eff49306219"
server_key_iv: "ba0883ae30d30c700a88b832a8a47804"
server_key_version: "encryptionkey@0"
I20260926 07:33:03.349402 5870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:33:03.378479 5870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:33:03.382649 5870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:33:03.382859 5870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:33:03.383495 5870 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:33:03.384411 5870 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:33:03.384471 5870 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:03.384518 5870 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:33:03.384569 5870 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:03.396306 5870 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.194:40153
I20260926 07:33:03.396358 5998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.194:40153 every 8 connection(s)
I20260926 07:33:03.397541 5870 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-1/data/info.pb
I20260926 07:33:03.406100 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 5870
I20260926 07:33:03.406288 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-1/wal/instance
I20260926 07:33:03.406620 22743 external_mini_cluster.cc:1428] Setting key 3a457d71a5de2b87f6e324d5631a4833
Sep 26 07:33:03 dist-test-slave-wdjc krb5kdc[5629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407983, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.194@KRBTEST.COM for kudu/127.22.53.254@KRBTEST.COM
I20260926 07:33:03.416041 5713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:03.399910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.194:42217 (local address 127.22.53.254:32891)
0926 07:33:03.400129 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:03.400134 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:03.401518 (+ 1384us) server_negotiation.cc:407] Connection header received
0926 07:33:03.403036 (+ 1518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:03.403039 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:03.403115 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:03.403227 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:03.405089 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:03.405659 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:03.407601 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:03.407854 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:03.407943 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:03.411296 (+ 3353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:03.411302 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:03.411310 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:03.411341 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:03.413408 (+ 2067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:03.414174 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:03.414180 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:03.414187 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:03.414273 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:03.414703 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:03.414706 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:03.414708 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:03.414951 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:03.415075 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:03.415695 (+ 620us) server_negotiation.cc:299] Negotiation successful
0926 07:33:03.415855 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260926 07:33:03.417152 6001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:03.400487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:32891 (local address 127.22.53.194:42217)
0926 07:33:03.401316 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:03.401372 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 07:33:03.402721 (+ 1349us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:33:03.403493 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:33:03.403502 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:33:03.403959 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:33:03.404867 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:03.404885 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:03.405894 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:03.405901 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:03.407397 (+ 1496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:03.407411 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:03.408065 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:03.408070 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:03.408359 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:03.409393 (+ 1034us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:33:03.409412 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:33:03.411050 (+ 1638us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:33:03.413634 (+ 2584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:33:03.413642 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:33:03.413666 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:33:03.414024 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:33:03.414385 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:33:03.414388 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:33:03.414390 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:33:03.414576 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:33:03.415077 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:33:03.415089 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:33:03.415404 (+ 315us) client_negotiation.cc:769] Sending connection context
0926 07:33:03.415640 (+ 236us) client_negotiation.cc:240] Negotiation successful
0926 07:33:03.416060 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":168,"threads_started":1}
WARNING: no policy specified for kudu/127.22.53.195@KRBTEST.COM; defaulting to no policy
I20260926 07:33:03.418876 5999 heartbeater.cc:347] Connected to a master server at 127.22.53.254:32891
I20260926 07:33:03.419232 5999 heartbeater.cc:464] Registering TS with master...
I20260926 07:33:03.419864 5999 heartbeater.cc:511] Master 127.22.53.254:32891 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.53.195@KRBTEST.COM" created.
I20260926 07:33:03.421489 5674 ts_manager.cc:194] Registered new tserver with Master: ed471121813d4d1a9eb3f4e1c02da65b (127.22.53.194:40153)
I20260926 07:33:03.422168 5674 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.53.194@KRBTEST.COM'} at 127.22.53.194:42217
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.53.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.195.keytab.
Entry for principal kudu/127.22.53.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.195.keytab.
I20260926 07:33:03.440732 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.53.195:0
--local_ip_for_outbound_sockets=127.22.53.195
--webserver_interface=127.22.53.195
--webserver_port=0
--tserver_master_addrs=127.22.53.254:32891
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.195.keytab
--principal=kudu/127.22.53.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:36403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:33:03.567447 6004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:33:03.567791 6004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:33:03.567845 6004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:33:03.572419 6004 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:33:03.572570 6004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:33:03.572645 6004 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.53.195
I20260926 07:33:03.578758 6004 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:36403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.195.keytab
--principal=kudu/127.22.53.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.53.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.53.195
--webserver_port=0
--tserver_master_addrs=127.22.53.254:32891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6004
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.53.195
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:33:03.580057 6004 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:33:03.581274 6004 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:33:03.592729 6012 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:33:03.592871 6004 server_base.cc:1034] running on GCE node
W20260926 07:33:03.592674 6010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:03.592674 6009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:33:03.593510 6004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:33:03.594383 6004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:33:03.595605 6004 hybrid_clock.cc:648] HybridClock initialized: now 1790407983595591 us; error 35 us; skew 500 ppm
Sep 26 07:33:03 dist-test-slave-wdjc krb5kdc[5629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407983, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:33:03.599563 6004 init.cc:377] Logged in from keytab as kudu/127.22.53.195@KRBTEST.COM (short username kudu)
I20260926 07:33:03.600836 6004 webserver.cc:460] Webserver started at http://127.22.53.195:43537/ using document root <none> and password file <none>
I20260926 07:33:03.601531 6004 fs_manager.cc:359] Metadata directory not provided
I20260926 07:33:03.601642 6004 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:33:03.601922 6004 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:33:03.604334 6004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-2/data/instance:
uuid: "5a0860bc51e14488b4f1810c7b69e9bd"
format_stamp: "Formatted at 2026-09-26 07:33:03 on dist-test-slave-wdjc"
server_key: "38343cbcbaa2c117f57ac9160cdf2edf"
server_key_iv: "9b0dc03585259ea20641300e57bba641"
server_key_version: "encryptionkey@0"
I20260926 07:33:03.605118 6004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-2/wal/instance:
uuid: "5a0860bc51e14488b4f1810c7b69e9bd"
format_stamp: "Formatted at 2026-09-26 07:33:03 on dist-test-slave-wdjc"
server_key: "38343cbcbaa2c117f57ac9160cdf2edf"
server_key_iv: "9b0dc03585259ea20641300e57bba641"
server_key_version: "encryptionkey@0"
I20260926 07:33:03.609706 6004 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 07:33:03.612788 6019 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:03.614828 6004 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 07:33:03.615072 6004 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-2/wal
uuid: "5a0860bc51e14488b4f1810c7b69e9bd"
format_stamp: "Formatted at 2026-09-26 07:33:03 on dist-test-slave-wdjc"
server_key: "38343cbcbaa2c117f57ac9160cdf2edf"
server_key_iv: "9b0dc03585259ea20641300e57bba641"
server_key_version: "encryptionkey@0"
I20260926 07:33:03.615252 6004 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:33:03.647140 6004 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:33:03.650857 6004 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:33:03.651054 6004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:33:03.651666 6004 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:33:03.652581 6004 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:33:03.652632 6004 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:03.652675 6004 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:33:03.652691 6004 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:03.664502 6004 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.195:35847
I20260926 07:33:03.665022 6132 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.195:35847 every 8 connection(s)
I20260926 07:33:03.665757 6004 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-2/data/info.pb
I20260926 07:33:03.669378 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 6004
I20260926 07:33:03.669566 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-2/wal/instance
I20260926 07:33:03.669909 22743 external_mini_cluster.cc:1428] Setting key 121e16969088eb3ddf50e33c26f504f5
Sep 26 07:33:03 dist-test-slave-wdjc krb5kdc[5629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407983, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.195@KRBTEST.COM for kudu/127.22.53.254@KRBTEST.COM
I20260926 07:33:03.682502 5713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:03.668173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.195:41497 (local address 127.22.53.254:32891)
0926 07:33:03.668354 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:03.668358 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:03.670229 (+ 1871us) server_negotiation.cc:407] Connection header received
0926 07:33:03.671561 (+ 1332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:03.671565 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:03.671625 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:03.671700 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:03.673274 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:03.673828 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:03.675119 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:03.675358 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:03.675487 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:03.678439 (+ 2952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:03.678443 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:03.678445 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:03.678474 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:03.680007 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:03.680756 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:03.680759 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:03.680761 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:03.680853 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:03.681252 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:03.681254 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:03.681256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:03.681466 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:03.681551 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:03.682223 (+ 672us) server_negotiation.cc:299] Negotiation successful
0926 07:33:03.682367 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 07:33:03.683530 6135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:03.668651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:32891 (local address 127.22.53.195:41497)
0926 07:33:03.669982 (+ 1331us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:03.670058 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0926 07:33:03.671273 (+ 1215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:33:03.671919 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:33:03.671927 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:33:03.672479 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:33:03.673107 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:03.673123 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:03.673981 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:03.673985 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:03.674961 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:03.674978 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:03.675543 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:03.675547 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:03.675754 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:03.676454 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:33:03.676472 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:33:03.678181 (+ 1709us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:33:03.680170 (+ 1989us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:33:03.680176 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:33:03.680193 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:33:03.680607 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:33:03.680958 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:33:03.680960 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:33:03.680962 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:33:03.681124 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:33:03.681572 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:33:03.681578 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:33:03.681892 (+ 314us) client_negotiation.cc:769] Sending connection context
0926 07:33:03.682083 (+ 191us) client_negotiation.cc:240] Negotiation successful
0926 07:33:03.682408 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1053,"thread_start_us":844,"threads_started":1}
I20260926 07:33:03.685110 6133 heartbeater.cc:347] Connected to a master server at 127.22.53.254:32891
I20260926 07:33:03.685417 6133 heartbeater.cc:464] Registering TS with master...
I20260926 07:33:03.686044 6133 heartbeater.cc:511] Master 127.22.53.254:32891 requested a full tablet report, sending...
I20260926 07:33:03.687372 5674 ts_manager.cc:194] Registered new tserver with Master: 5a0860bc51e14488b4f1810c7b69e9bd (127.22.53.195:35847)
I20260926 07:33:03.688024 5674 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.53.195@KRBTEST.COM'} at 127.22.53.195:41497
I20260926 07:33:03.696971 22743 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:33:03.707823 5713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:03.699258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47690 (local address 127.22.53.254:32891)
0926 07:33:03.699456 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:03.699460 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:03.699595 (+ 135us) server_negotiation.cc:407] Connection header received
0926 07:33:03.699790 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:03.699793 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:03.699849 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:03.699943 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:03.700848 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:03.701402 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:03.702340 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:03.702573 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:03.702691 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:03.704271 (+ 1580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:03.704274 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:03.704277 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:03.704308 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:03.705830 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:03.706372 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:03.706381 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:03.706383 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:03.706472 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:03.706775 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:03.706778 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:03.706780 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:03.707064 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:03.707215 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:03.707492 (+ 277us) server_negotiation.cc:299] Negotiation successful
0926 07:33:03.707621 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260926 07:33:03.711659 5674 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47690:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:33:03.715240 5674 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:33:03.733381 6145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:03.726161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.195:35847 (local address 127.0.0.1:60066)
0926 07:33:03.727368 (+ 1207us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:03.727383 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:33:03.727477 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:33:03.728285 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:33:03.728289 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:33:03.728309 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:33:03.728434 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:03.728441 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:03.729943 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:03.729947 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:03.731343 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:03.731356 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:03.732794 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:03.732798 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:03.732900 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:03.732966 (+ 66us) client_negotiation.cc:769] Sending connection context
0926 07:33:03.733109 (+ 143us) client_negotiation.cc:240] Negotiation successful
0926 07:33:03.733238 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1139,"mutex_wait_us":60,"thread_start_us":168,"threads_started":1}
I20260926 07:33:03.733511 6146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:03.726449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60066 (local address 127.22.53.195:35847)
0926 07:33:03.727774 (+ 1325us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:03.727788 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:03.727812 (+ 24us) server_negotiation.cc:407] Connection header received
0926 07:33:03.727890 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:03.727897 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:03.728101 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:03.728606 (+ 505us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:33:03.728797 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:03.729748 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:03.731465 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:03.732619 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:03.732856 (+ 237us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:03.733017 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:03.733152 (+ 135us) server_negotiation.cc:299] Negotiation successful
0926 07:33:03.733274 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1199,"thread_start_us":141,"threads_started":1}
I20260926 07:33:03.734328 6148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:03.727397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.194:40153 (local address 127.0.0.1:47724)
0926 07:33:03.727909 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:03.727925 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:33:03.728021 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:33:03.728998 (+ 977us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:33:03.729001 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:33:03.729019 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:33:03.729121 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:03.729127 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:03.730560 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:03.730563 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:03.731596 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:03.731607 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:03.732675 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:03.732679 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:03.733937 (+ 1258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:03.733988 (+ 51us) client_negotiation.cc:769] Sending connection context
0926 07:33:03.734079 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 07:33:03.734171 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"mutex_wait_us":40,"thread_start_us":112,"threads_started":1}
I20260926 07:33:03.734458 6149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:03.727860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47724 (local address 127.22.53.194:40153)
0926 07:33:03.728330 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:03.728342 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:03.728363 (+ 21us) server_negotiation.cc:407] Connection header received
0926 07:33:03.728439 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:03.728444 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:03.728677 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:03.728827 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:33:03.729264 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:03.730418 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:03.731737 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:03.732513 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:03.732671 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:03.732870 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:03.734158 (+ 1288us) server_negotiation.cc:299] Negotiation successful
0926 07:33:03.734282 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":112,"threads_started":1}
I20260926 07:33:03.735262 6144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:03.726027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.193:40583 (local address 127.0.0.1:56048)
0926 07:33:03.727200 (+ 1173us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:03.727282 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0926 07:33:03.727436 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:33:03.728159 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:33:03.728165 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:33:03.728192 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:33:03.728360 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:03.728367 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:03.729888 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:03.729891 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:03.732667 (+ 2776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:03.732681 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:03.734708 (+ 2027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:03.734716 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:03.734842 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:03.734864 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 07:33:03.735004 (+ 140us) client_negotiation.cc:240] Negotiation successful
0926 07:33:03.735111 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":973,"mutex_wait_us":23,"thread_start_us":105,"threads_started":1}
I20260926 07:33:03.735342 6147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:03.726435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56048 (local address 127.22.53.193:40583)
0926 07:33:03.727158 (+ 723us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:03.727165 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:03.727602 (+ 437us) server_negotiation.cc:407] Connection header received
0926 07:33:03.727693 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:03.727699 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:03.727962 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:03.728121 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:33:03.728492 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:03.729716 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:03.732846 (+ 3130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:03.734005 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:03.734157 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:03.734323 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:03.735051 (+ 728us) server_negotiation.cc:299] Negotiation successful
0926 07:33:03.735173 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":615,"thread_start_us":158,"threads_started":1}
I20260926 07:33:03.737776 6067 tablet_service.cc:1467] Processing CreateTablet for tablet e514e01e6b1e4f0aabf9b2b562359829 (DEFAULT_TABLE table=test-table [id=2194e33bd08a4ad88b469277d6318869]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:33:03.737780 5933 tablet_service.cc:1467] Processing CreateTablet for tablet e514e01e6b1e4f0aabf9b2b562359829 (DEFAULT_TABLE table=test-table [id=2194e33bd08a4ad88b469277d6318869]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:33:03.739053 6067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e514e01e6b1e4f0aabf9b2b562359829. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:33:03.739060 5933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e514e01e6b1e4f0aabf9b2b562359829. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:33:03.746315 5799 tablet_service.cc:1467] Processing CreateTablet for tablet e514e01e6b1e4f0aabf9b2b562359829 (DEFAULT_TABLE table=test-table [id=2194e33bd08a4ad88b469277d6318869]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:33:03.746944 6150 tablet_bootstrap.cc:492] T e514e01e6b1e4f0aabf9b2b562359829 P 5a0860bc51e14488b4f1810c7b69e9bd: Bootstrap starting.
I20260926 07:33:03.747694 5799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e514e01e6b1e4f0aabf9b2b562359829. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:33:03.748463 6151 tablet_bootstrap.cc:492] T e514e01e6b1e4f0aabf9b2b562359829 P ed471121813d4d1a9eb3f4e1c02da65b: Bootstrap starting.
I20260926 07:33:03.749617 6150 tablet_bootstrap.cc:654] T e514e01e6b1e4f0aabf9b2b562359829 P 5a0860bc51e14488b4f1810c7b69e9bd: Neither blocks nor log segments found. Creating new log.
I20260926 07:33:03.751375 6150 log.cc:824] T e514e01e6b1e4f0aabf9b2b562359829 P 5a0860bc51e14488b4f1810c7b69e9bd: Log is configured to *not* fsync() on all Append() calls
I20260926 07:33:03.751524 6151 tablet_bootstrap.cc:654] T e514e01e6b1e4f0aabf9b2b562359829 P ed471121813d4d1a9eb3f4e1c02da65b: Neither blocks nor log segments found. Creating new log.
I20260926 07:33:03.752625 6151 log.cc:824] T e514e01e6b1e4f0aabf9b2b562359829 P ed471121813d4d1a9eb3f4e1c02da65b: Log is configured to *not* fsync() on all Append() calls
I20260926 07:33:03.753990 6150 tablet_bootstrap.cc:492] T e514e01e6b1e4f0aabf9b2b562359829 P 5a0860bc51e14488b4f1810c7b69e9bd: No bootstrap required, opened a new log
I20260926 07:33:03.754259 6150 ts_tablet_manager.cc:1397] T e514e01e6b1e4f0aabf9b2b562359829 P 5a0860bc51e14488b4f1810c7b69e9bd: Time spent bootstrapping tablet: real 0.007s user 0.007s sys 0.000s
I20260926 07:33:03.756270 6151 tablet_bootstrap.cc:492] T e514e01e6b1e4f0aabf9b2b562359829 P ed471121813d4d1a9eb3f4e1c02da65b: No bootstrap required, opened a new log
I20260926 07:33:03.756520 6151 ts_tablet_manager.cc:1397] T e514e01e6b1e4f0aabf9b2b562359829 P ed471121813d4d1a9eb3f4e1c02da65b: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260926 07:33:03.756608 6154 tablet_bootstrap.cc:492] T e514e01e6b1e4f0aabf9b2b562359829 P 35ab6a8afa1a4639b35923c7d881a6c7: Bootstrap starting.
I20260926 07:33:03.759123 6150 raft_consensus.cc:348] T e514e01e6b1e4f0aabf9b2b562359829 P 5a0860bc51e14488b4f1810c7b69e9bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ab6a8afa1a4639b35923c7d881a6c7" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 40583 } } peers { permanent_uuid: "5a0860bc51e14488b4f1810c7b69e9bd" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 35847 } } peers { permanent_uuid: "ed471121813d4d1a9eb3f4e1c02da65b" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 40153 } }
I20260926 07:33:03.759435 6154 tablet_bootstrap.cc:654] T e514e01e6b1e4f0aabf9b2b562359829 P 35ab6a8afa1a4639b35923c7d881a6c7: Neither blocks nor log segments found. Creating new log.
I20260926 07:33:03.759500 6150 raft_consensus.cc:374] T e514e01e6b1e4f0aabf9b2b562359829 P 5a0860bc51e14488b4f1810c7b69e9bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:33:03.759554 6150 raft_consensus.cc:729] T e514e01e6b1e4f0aabf9b2b562359829 P 5a0860bc51e14488b4f1810c7b69e9bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a0860bc51e14488b4f1810c7b69e9bd, State: Initialized, Role: FOLLOWER
I20260926 07:33:03.760102 6150 consensus_queue.cc:260] T e514e01e6b1e4f0aabf9b2b562359829 P 5a0860bc51e14488b4f1810c7b69e9bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ab6a8afa1a4639b35923c7d881a6c7" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 40583 } } peers { permanent_uuid: "5a0860bc51e14488b4f1810c7b69e9bd" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 35847 } } peers { permanent_uuid: "ed471121813d4d1a9eb3f4e1c02da65b" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 40153 } }
I20260926 07:33:03.760563 6154 log.cc:824] T e514e01e6b1e4f0aabf9b2b562359829 P 35ab6a8afa1a4639b35923c7d881a6c7: Log is configured to *not* fsync() on all Append() calls
I20260926 07:33:03.760838 6151 raft_consensus.cc:348] T e514e01e6b1e4f0aabf9b2b562359829 P ed471121813d4d1a9eb3f4e1c02da65b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ab6a8afa1a4639b35923c7d881a6c7" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 40583 } } peers { permanent_uuid: "5a0860bc51e14488b4f1810c7b69e9bd" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 35847 } } peers { permanent_uuid: "ed471121813d4d1a9eb3f4e1c02da65b" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 40153 } }
I20260926 07:33:03.761076 6133 heartbeater.cc:503] Master 127.22.53.254:32891 was elected leader, sending a full tablet report...
I20260926 07:33:03.761140 6151 raft_consensus.cc:374] T e514e01e6b1e4f0aabf9b2b562359829 P ed471121813d4d1a9eb3f4e1c02da65b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:33:03.761217 6151 raft_consensus.cc:729] T e514e01e6b1e4f0aabf9b2b562359829 P ed471121813d4d1a9eb3f4e1c02da65b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed471121813d4d1a9eb3f4e1c02da65b, State: Initialized, Role: FOLLOWER
I20260926 07:33:03.761389 6150 ts_tablet_manager.cc:1428] T e514e01e6b1e4f0aabf9b2b562359829 P 5a0860bc51e14488b4f1810c7b69e9bd: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260926 07:33:03.761760 6151 consensus_queue.cc:260] T e514e01e6b1e4f0aabf9b2b562359829 P ed471121813d4d1a9eb3f4e1c02da65b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ab6a8afa1a4639b35923c7d881a6c7" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 40583 } } peers { permanent_uuid: "5a0860bc51e14488b4f1810c7b69e9bd" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 35847 } } peers { permanent_uuid: "ed471121813d4d1a9eb3f4e1c02da65b" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 40153 } }
I20260926 07:33:03.762843 6151 ts_tablet_manager.cc:1428] T e514e01e6b1e4f0aabf9b2b562359829 P ed471121813d4d1a9eb3f4e1c02da65b: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260926 07:33:03.763306 6154 tablet_bootstrap.cc:492] T e514e01e6b1e4f0aabf9b2b562359829 P 35ab6a8afa1a4639b35923c7d881a6c7: No bootstrap required, opened a new log
I20260926 07:33:03.763561 6154 ts_tablet_manager.cc:1397] T e514e01e6b1e4f0aabf9b2b562359829 P 35ab6a8afa1a4639b35923c7d881a6c7: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260926 07:33:03.765026 5999 heartbeater.cc:503] Master 127.22.53.254:32891 was elected leader, sending a full tablet report...
I20260926 07:33:03.768854 6154 raft_consensus.cc:348] T e514e01e6b1e4f0aabf9b2b562359829 P 35ab6a8afa1a4639b35923c7d881a6c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ab6a8afa1a4639b35923c7d881a6c7" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 40583 } } peers { permanent_uuid: "5a0860bc51e14488b4f1810c7b69e9bd" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 35847 } } peers { permanent_uuid: "ed471121813d4d1a9eb3f4e1c02da65b" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 40153 } }
I20260926 07:33:03.769270 6154 raft_consensus.cc:374] T e514e01e6b1e4f0aabf9b2b562359829 P 35ab6a8afa1a4639b35923c7d881a6c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:33:03.769659 6154 raft_consensus.cc:729] T e514e01e6b1e4f0aabf9b2b562359829 P 35ab6a8afa1a4639b35923c7d881a6c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35ab6a8afa1a4639b35923c7d881a6c7, State: Initialized, Role: FOLLOWER
I20260926 07:33:03.770910 6154 consensus_queue.cc:260] T e514e01e6b1e4f0aabf9b2b562359829 P 35ab6a8afa1a4639b35923c7d881a6c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ab6a8afa1a4639b35923c7d881a6c7" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 40583 } } peers { permanent_uuid: "5a0860bc51e14488b4f1810c7b69e9bd" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 35847 } } peers { permanent_uuid: "ed471121813d4d1a9eb3f4e1c02da65b" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 40153 } }
I20260926 07:33:03.771991 5865 heartbeater.cc:503] Master 127.22.53.254:32891 was elected leader, sending a full tablet report...
I20260926 07:33:03.772379 6154 ts_tablet_manager.cc:1428] T e514e01e6b1e4f0aabf9b2b562359829 P 35ab6a8afa1a4639b35923c7d881a6c7: Time spent starting tablet: real 0.009s user 0.009s sys 0.000s
I20260926 07:33:03.885414 6157 raft_consensus.cc:482] T e514e01e6b1e4f0aabf9b2b562359829 P ed471121813d4d1a9eb3f4e1c02da65b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:33:03.885686 6157 raft_consensus.cc:504] T e514e01e6b1e4f0aabf9b2b562359829 P ed471121813d4d1a9eb3f4e1c02da65b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ab6a8afa1a4639b35923c7d881a6c7" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 40583 } } peers { permanent_uuid: "5a0860bc51e14488b4f1810c7b69e9bd" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 35847 } } peers { permanent_uuid: "ed471121813d4d1a9eb3f4e1c02da65b" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 40153 } }
I20260926 07:33:03.887127 6157 leader_election.cc:288] T e514e01e6b1e4f0aabf9b2b562359829 P ed471121813d4d1a9eb3f4e1c02da65b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 35ab6a8afa1a4639b35923c7d881a6c7 (127.22.53.193:40583), 5a0860bc51e14488b4f1810c7b69e9bd (127.22.53.195:35847)
W20260926 07:33:03.888885 5866 tablet.cc:2369] T e514e01e6b1e4f0aabf9b2b562359829 P 35ab6a8afa1a4639b35923c7d881a6c7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:33:03.893240 6147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:03.887634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.194:36819 (local address 127.22.53.193:40583)
0926 07:33:03.887792 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:03.887799 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:03.887815 (+ 16us) server_negotiation.cc:407] Connection header received
0926 07:33:03.887881 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:03.887884 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:03.887936 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:03.888026 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:33:03.888655 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:03.889464 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:03.891598 (+ 2134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:03.892367 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:03.892575 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:03.892644 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:03.892999 (+ 355us) server_negotiation.cc:299] Negotiation successful
0926 07:33:03.893130 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 07:33:03.893239 6146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:03.887678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.194:49155 (local address 127.22.53.195:35847)
0926 07:33:03.887823 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:03.887831 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:03.888038 (+ 207us) server_negotiation.cc:407] Connection header received
0926 07:33:03.888109 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:03.888111 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:03.888160 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:03.888282 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:33:03.889144 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:03.889917 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:03.891706 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:03.892374 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:03.892575 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:03.892644 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:03.892999 (+ 355us) server_negotiation.cc:299] Negotiation successful
0926 07:33:03.893091 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 07:33:03.893268 6001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:03.887396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.193:40583 (local address 127.22.53.194:36819)
0926 07:33:03.887564 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:03.887582 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 07:33:03.887695 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:33:03.888075 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:33:03.888080 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:33:03.888108 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:33:03.888272 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:03.888281 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:03.889619 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:03.889623 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:03.891288 (+ 1665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:03.891306 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:03.892581 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:03.892586 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:03.892822 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:03.892852 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 07:33:03.892945 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 07:33:03.893075 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63,"mutex_wait_us":96}
I20260926 07:33:03.893276 6159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:03.887527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.195:35847 (local address 127.22.53.194:49155)
0926 07:33:03.887918 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:03.887932 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:33:03.888034 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:33:03.888302 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:33:03.888305 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:33:03.888320 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:33:03.888772 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:03.888869 (+ 97us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:03.890270 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:03.890275 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:03.891482 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:03.891490 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:03.892581 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:03.892586 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:03.892777 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:03.892852 (+ 75us) client_negotiation.cc:769] Sending connection context
0926 07:33:03.892945 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 07:33:03.893075 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"mutex_wait_us":82,"thread_start_us":118,"threads_started":1}
I20260926 07:33:03.894134 5819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e514e01e6b1e4f0aabf9b2b562359829" candidate_uuid: "ed471121813d4d1a9eb3f4e1c02da65b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35ab6a8afa1a4639b35923c7d881a6c7" is_pre_election: true
I20260926 07:33:03.894287 6087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e514e01e6b1e4f0aabf9b2b562359829" candidate_uuid: "ed471121813d4d1a9eb3f4e1c02da65b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a0860bc51e14488b4f1810c7b69e9bd" is_pre_election: true
I20260926 07:33:03.894532 5819 raft_consensus.cc:2454] T e514e01e6b1e4f0aabf9b2b562359829 P 35ab6a8afa1a4639b35923c7d881a6c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed471121813d4d1a9eb3f4e1c02da65b in term 0.
I20260926 07:33:03.894634 6087 raft_consensus.cc:2454] T e514e01e6b1e4f0aabf9b2b562359829 P 5a0860bc51e14488b4f1810c7b69e9bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed471121813d4d1a9eb3f4e1c02da65b in term 0.
I20260926 07:33:03.895203 5887 leader_election.cc:302] T e514e01e6b1e4f0aabf9b2b562359829 P ed471121813d4d1a9eb3f4e1c02da65b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 35ab6a8afa1a4639b35923c7d881a6c7, 5a0860bc51e14488b4f1810c7b69e9bd, ed471121813d4d1a9eb3f4e1c02da65b; no voters:
I20260926 07:33:03.895546 6157 raft_consensus.cc:2781] T e514e01e6b1e4f0aabf9b2b562359829 P ed471121813d4d1a9eb3f4e1c02da65b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:33:03.895645 6157 raft_consensus.cc:482] T e514e01e6b1e4f0aabf9b2b562359829 P ed471121813d4d1a9eb3f4e1c02da65b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:33:03.895694 6157 raft_consensus.cc:3037] T e514e01e6b1e4f0aabf9b2b562359829 P ed471121813d4d1a9eb3f4e1c02da65b [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:33:03.896983 6157 raft_consensus.cc:504] T e514e01e6b1e4f0aabf9b2b562359829 P ed471121813d4d1a9eb3f4e1c02da65b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ab6a8afa1a4639b35923c7d881a6c7" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 40583 } } peers { permanent_uuid: "5a0860bc51e14488b4f1810c7b69e9bd" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 35847 } } peers { permanent_uuid: "ed471121813d4d1a9eb3f4e1c02da65b" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 40153 } }
I20260926 07:33:03.897434 6157 leader_election.cc:288] T e514e01e6b1e4f0aabf9b2b562359829 P ed471121813d4d1a9eb3f4e1c02da65b [CANDIDATE]: Term 1 election: Requested vote from peers 35ab6a8afa1a4639b35923c7d881a6c7 (127.22.53.193:40583), 5a0860bc51e14488b4f1810c7b69e9bd (127.22.53.195:35847)
I20260926 07:33:03.897938 6087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e514e01e6b1e4f0aabf9b2b562359829" candidate_uuid: "ed471121813d4d1a9eb3f4e1c02da65b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a0860bc51e14488b4f1810c7b69e9bd"
I20260926 07:33:03.898118 6087 raft_consensus.cc:3037] T e514e01e6b1e4f0aabf9b2b562359829 P 5a0860bc51e14488b4f1810c7b69e9bd [term 0 FOLLOWER]: Advancing to term 1
W20260926 07:33:03.899037 6000 tablet.cc:2369] T e514e01e6b1e4f0aabf9b2b562359829 P ed471121813d4d1a9eb3f4e1c02da65b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:33:03.899516 6087 raft_consensus.cc:2454] T e514e01e6b1e4f0aabf9b2b562359829 P 5a0860bc51e14488b4f1810c7b69e9bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed471121813d4d1a9eb3f4e1c02da65b in term 1.
I20260926 07:33:03.899994 5888 leader_election.cc:302] T e514e01e6b1e4f0aabf9b2b562359829 P ed471121813d4d1a9eb3f4e1c02da65b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a0860bc51e14488b4f1810c7b69e9bd, ed471121813d4d1a9eb3f4e1c02da65b; no voters:
I20260926 07:33:03.899962 5819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e514e01e6b1e4f0aabf9b2b562359829" candidate_uuid: "ed471121813d4d1a9eb3f4e1c02da65b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35ab6a8afa1a4639b35923c7d881a6c7"
I20260926 07:33:03.900128 5819 raft_consensus.cc:3037] T e514e01e6b1e4f0aabf9b2b562359829 P 35ab6a8afa1a4639b35923c7d881a6c7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:33:03.900226 6157 raft_consensus.cc:2781] T e514e01e6b1e4f0aabf9b2b562359829 P ed471121813d4d1a9eb3f4e1c02da65b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:33:03.900476 6157 raft_consensus.cc:686] T e514e01e6b1e4f0aabf9b2b562359829 P ed471121813d4d1a9eb3f4e1c02da65b [term 1 LEADER]: Becoming Leader. State: Replica: ed471121813d4d1a9eb3f4e1c02da65b, State: Running, Role: LEADER
I20260926 07:33:03.900914 6157 consensus_queue.cc:237] T e514e01e6b1e4f0aabf9b2b562359829 P ed471121813d4d1a9eb3f4e1c02da65b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ab6a8afa1a4639b35923c7d881a6c7" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 40583 } } peers { permanent_uuid: "5a0860bc51e14488b4f1810c7b69e9bd" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 35847 } } peers { permanent_uuid: "ed471121813d4d1a9eb3f4e1c02da65b" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 40153 } }
I20260926 07:33:03.901675 5819 raft_consensus.cc:2454] T e514e01e6b1e4f0aabf9b2b562359829 P 35ab6a8afa1a4639b35923c7d881a6c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed471121813d4d1a9eb3f4e1c02da65b in term 1.
I20260926 07:33:03.905277 5673 catalog_manager.cc:5495] T e514e01e6b1e4f0aabf9b2b562359829 P ed471121813d4d1a9eb3f4e1c02da65b reported cstate change: term changed from 0 to 1, leader changed from <none> to ed471121813d4d1a9eb3f4e1c02da65b (127.22.53.194). New cstate: current_term: 1 leader_uuid: "ed471121813d4d1a9eb3f4e1c02da65b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ab6a8afa1a4639b35923c7d881a6c7" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 40583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a0860bc51e14488b4f1810c7b69e9bd" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 35847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed471121813d4d1a9eb3f4e1c02da65b" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 40153 } health_report { overall_health: HEALTHY } } }
W20260926 07:33:03.917073 6134 tablet.cc:2369] T e514e01e6b1e4f0aabf9b2b562359829 P 5a0860bc51e14488b4f1810c7b69e9bd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:33:03.931507 6149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:03.926158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47726 (local address 127.22.53.194:40153)
0926 07:33:03.926346 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:03.926350 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:03.926398 (+ 48us) server_negotiation.cc:407] Connection header received
0926 07:33:03.926596 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:03.926599 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:03.926666 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:03.926784 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:33:03.927086 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:03.927707 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:03.928834 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:03.929189 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:03.929302 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:03.929721 (+ 419us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:33:03.930647 (+ 926us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:33:03.930850 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:03.931157 (+ 307us) server_negotiation.cc:299] Negotiation successful
0926 07:33:03.931292 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260926 07:33:03.939072 5819 raft_consensus.cc:1261] T e514e01e6b1e4f0aabf9b2b562359829 P 35ab6a8afa1a4639b35923c7d881a6c7 [term 1 FOLLOWER]: Refusing update from remote peer ed471121813d4d1a9eb3f4e1c02da65b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:33:03.939080 6087 raft_consensus.cc:1261] T e514e01e6b1e4f0aabf9b2b562359829 P 5a0860bc51e14488b4f1810c7b69e9bd [term 1 FOLLOWER]: Refusing update from remote peer ed471121813d4d1a9eb3f4e1c02da65b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:33:03.939915 6157 consensus_queue.cc:1035] T e514e01e6b1e4f0aabf9b2b562359829 P ed471121813d4d1a9eb3f4e1c02da65b [LEADER]: Connected to new peer: Peer: permanent_uuid: "35ab6a8afa1a4639b35923c7d881a6c7" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 40583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:33:03.940194 6160 consensus_queue.cc:1035] T e514e01e6b1e4f0aabf9b2b562359829 P ed471121813d4d1a9eb3f4e1c02da65b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a0860bc51e14488b4f1810c7b69e9bd" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 35847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:33:03.949011 6163 mvcc.cc:205] Tried to move back new op lower bound from 7333511102205521920 to 7333511102061965312. Current Snapshot: MvccSnapshot[applied={T|T < 7333511102205521920}]
I20260926 07:33:03.963424 5673 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47690:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:33:03.963696 5673 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47690:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:33:03.967626 5673 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 788e7a47032d4f458d5493fcf0021f9d: Sending DeleteTablet for 3 replicas of tablet e514e01e6b1e4f0aabf9b2b562359829
I20260926 07:33:03.968796 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 5736
I20260926 07:33:03.968742 6067 tablet_service.cc:1514] Processing DeleteTablet for tablet e514e01e6b1e4f0aabf9b2b562359829 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:33:03 UTC) from {username='kudu', principal='kudu/127.22.53.254@KRBTEST.COM'} at 127.0.0.1:60066
I20260926 07:33:03.968770 5799 tablet_service.cc:1514] Processing DeleteTablet for tablet e514e01e6b1e4f0aabf9b2b562359829 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:33:03 UTC) from {username='kudu', principal='kudu/127.22.53.254@KRBTEST.COM'} at 127.0.0.1:56048
I20260926 07:33:03.969849 5933 tablet_service.cc:1514] Processing DeleteTablet for tablet e514e01e6b1e4f0aabf9b2b562359829 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:33:03 UTC) from {username='kudu', principal='kudu/127.22.53.254@KRBTEST.COM'} at 127.0.0.1:47724
I20260926 07:33:03.970806 6174 tablet_replica.cc:330] T e514e01e6b1e4f0aabf9b2b562359829 P ed471121813d4d1a9eb3f4e1c02da65b: stopping tablet replica
I20260926 07:33:03.971186 6174 raft_consensus.cc:2229] T e514e01e6b1e4f0aabf9b2b562359829 P ed471121813d4d1a9eb3f4e1c02da65b [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:33:03.971673 6174 raft_consensus.cc:2258] T e514e01e6b1e4f0aabf9b2b562359829 P ed471121813d4d1a9eb3f4e1c02da65b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:33:03.972195 6173 tablet_replica.cc:330] T e514e01e6b1e4f0aabf9b2b562359829 P 5a0860bc51e14488b4f1810c7b69e9bd: stopping tablet replica
I20260926 07:33:03.972582 6173 raft_consensus.cc:2229] T e514e01e6b1e4f0aabf9b2b562359829 P 5a0860bc51e14488b4f1810c7b69e9bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:33:03.972973 6173 raft_consensus.cc:2258] T e514e01e6b1e4f0aabf9b2b562359829 P 5a0860bc51e14488b4f1810c7b69e9bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:33:03.974021 6174 ts_tablet_manager.cc:1905] T e514e01e6b1e4f0aabf9b2b562359829 P ed471121813d4d1a9eb3f4e1c02da65b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:33:03.974645 6173 ts_tablet_manager.cc:1905] T e514e01e6b1e4f0aabf9b2b562359829 P 5a0860bc51e14488b4f1810c7b69e9bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:33:03.977520 6174 ts_tablet_manager.cc:1918] T e514e01e6b1e4f0aabf9b2b562359829 P ed471121813d4d1a9eb3f4e1c02da65b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:33:03.977520 6173 ts_tablet_manager.cc:1918] T e514e01e6b1e4f0aabf9b2b562359829 P 5a0860bc51e14488b4f1810c7b69e9bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:33:03.977669 6173 log.cc:1196] T e514e01e6b1e4f0aabf9b2b562359829 P 5a0860bc51e14488b4f1810c7b69e9bd: Deleting WAL directory at /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-2/wal/wals/e514e01e6b1e4f0aabf9b2b562359829
I20260926 07:33:03.977854 6174 log.cc:1196] T e514e01e6b1e4f0aabf9b2b562359829 P ed471121813d4d1a9eb3f4e1c02da65b: Deleting WAL directory at /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-1/wal/wals/e514e01e6b1e4f0aabf9b2b562359829
I20260926 07:33:03.978063 6173 ts_tablet_manager.cc:1939] T e514e01e6b1e4f0aabf9b2b562359829 P 5a0860bc51e14488b4f1810c7b69e9bd: Deleting consensus metadata
I20260926 07:33:03.978374 6174 ts_tablet_manager.cc:1939] T e514e01e6b1e4f0aabf9b2b562359829 P ed471121813d4d1a9eb3f4e1c02da65b: Deleting consensus metadata
W20260926 07:33:03.978865 5659 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.53.193:40583 (error 108)
I20260926 07:33:03.979743 5658 catalog_manager.cc:4841] TS ed471121813d4d1a9eb3f4e1c02da65b (127.22.53.194:40153): tablet e514e01e6b1e4f0aabf9b2b562359829 (table test-table [id=2194e33bd08a4ad88b469277d6318869]) successfully deleted
I20260926 07:33:03.980055 5660 catalog_manager.cc:4841] TS 5a0860bc51e14488b4f1810c7b69e9bd (127.22.53.195:35847): tablet e514e01e6b1e4f0aabf9b2b562359829 (table test-table [id=2194e33bd08a4ad88b469277d6318869]) successfully deleted
I20260926 07:33:03.980355 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 5870
I20260926 07:33:03.980793 6144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:03.980393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.193:40583 (local address 127.0.0.1:56056)
0926 07:33:03.980581 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:03.980672 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.193:40583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260926 07:33:03.981232 5659 catalog_manager.cc:4584] TS 35ab6a8afa1a4639b35923c7d881a6c7 (127.22.53.193:40583): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e514e01e6b1e4f0aabf9b2b562359829: Network error: Client connection negotiation failed: client connection to 127.22.53.193:40583: connect: Connection refused (error 111)
I20260926 07:33:03.987321 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 6004
I20260926 07:33:03.994113 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 5642
I20260926 07:33:24.002070 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.53.254:32891
--webserver_interface=127.22.53.254
--webserver_port=37697
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab
--principal=kudu/127.22.53.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:36403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.53.254:32891
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:33:24.128715 6176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:33:24.129124 6176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:33:24.129174 6176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:33:24.133594 6176 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 07:33:24.133689 6176 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 07:33:24.133706 6176 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 07:33:24.133721 6176 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:33:24.133742 6176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:33:24.133761 6176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:33:24.133821 6176 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:33:24.133849 6176 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:33:24.139932 6176 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:36403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.22.53.254:32891
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab
--principal=kudu/127.22.53.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.53.254:32891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.53.254
--webserver_port=37697
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6176
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:33:24.141255 6176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:33:24.142421 6176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:33:24.157562 6182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:24.157562 6185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:33:24.157636 6176 server_base.cc:1034] running on GCE node
W20260926 07:33:24.157562 6183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:33:24.158538 6176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:33:24.159967 6176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:33:24.161202 6176 hybrid_clock.cc:648] HybridClock initialized: now 1790408004161173 us; error 53 us; skew 500 ppm
Sep 26 07:33:24 dist-test-slave-wdjc krb5kdc[5629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408004, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:33:24.171166 6176 init.cc:377] Logged in from keytab as kudu/127.22.53.254@KRBTEST.COM (short username kudu)
I20260926 07:33:24.173151 6176 webserver.cc:460] Webserver started at http://127.22.53.254:37697/ using document root <none> and password file <none>
I20260926 07:33:24.174190 6176 fs_manager.cc:359] Metadata directory not provided
I20260926 07:33:24.174439 6176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:33:24.179260 6176 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 07:33:24.182173 6192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:24.184082 6176 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 07:33:24.184297 6176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/master-0/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/master-0/wal
uuid: "788e7a47032d4f458d5493fcf0021f9d"
format_stamp: "Formatted at 2026-09-26 07:33:02 on dist-test-slave-wdjc"
server_key: "299243d4b6966ad5336afd5e0b3334a4"
server_key_iv: "6e2a817fc2ea3438e7c194a94f3aabfa"
server_key_version: "encryptionkey@0"
I20260926 07:33:24.185158 6176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:33:24.202518 6176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:33:24.206287 6176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:33:24.206534 6176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:33:24.214733 6176 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.254:32891
I20260926 07:33:24.215078 6244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.254:32891 every 8 connection(s)
I20260926 07:33:24.216014 6176 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
I20260926 07:33:24.220886 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 6176
I20260926 07:33:24.221211 6245 sys_catalog.cc:263] Verifying existing consensus state
I20260926 07:33:24.221416 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.53.193:40583
--local_ip_for_outbound_sockets=127.22.53.193
--tserver_master_addrs=127.22.53.254:32891
--webserver_port=46431
--webserver_interface=127.22.53.193
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.193.keytab
--principal=kudu/127.22.53.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:36403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:33:24.223519 6245 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 788e7a47032d4f458d5493fcf0021f9d: Bootstrap starting.
I20260926 07:33:24.245512 6245 log.cc:824] T 00000000000000000000000000000000 P 788e7a47032d4f458d5493fcf0021f9d: Log is configured to *not* fsync() on all Append() calls
I20260926 07:33:24.254864 6245 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 788e7a47032d4f458d5493fcf0021f9d: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 07:33:24.255362 6245 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 788e7a47032d4f458d5493fcf0021f9d: Bootstrap complete.
I20260926 07:33:24.261938 6245 raft_consensus.cc:348] T 00000000000000000000000000000000 P 788e7a47032d4f458d5493fcf0021f9d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788e7a47032d4f458d5493fcf0021f9d" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 32891 } }
I20260926 07:33:24.262419 6245 raft_consensus.cc:729] T 00000000000000000000000000000000 P 788e7a47032d4f458d5493fcf0021f9d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 788e7a47032d4f458d5493fcf0021f9d, State: Initialized, Role: FOLLOWER
I20260926 07:33:24.262974 6245 consensus_queue.cc:260] T 00000000000000000000000000000000 P 788e7a47032d4f458d5493fcf0021f9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788e7a47032d4f458d5493fcf0021f9d" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 32891 } }
I20260926 07:33:24.263100 6245 raft_consensus.cc:388] T 00000000000000000000000000000000 P 788e7a47032d4f458d5493fcf0021f9d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:33:24.263204 6245 raft_consensus.cc:482] T 00000000000000000000000000000000 P 788e7a47032d4f458d5493fcf0021f9d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:33:24.263341 6245 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 788e7a47032d4f458d5493fcf0021f9d [term 1 FOLLOWER]: Advancing to term 2
I20260926 07:33:24.265646 6245 raft_consensus.cc:504] T 00000000000000000000000000000000 P 788e7a47032d4f458d5493fcf0021f9d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788e7a47032d4f458d5493fcf0021f9d" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 32891 } }
I20260926 07:33:24.266053 6245 leader_election.cc:302] T 00000000000000000000000000000000 P 788e7a47032d4f458d5493fcf0021f9d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 788e7a47032d4f458d5493fcf0021f9d; no voters:
I20260926 07:33:24.266458 6245 leader_election.cc:288] T 00000000000000000000000000000000 P 788e7a47032d4f458d5493fcf0021f9d [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 07:33:24.266564 6249 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 788e7a47032d4f458d5493fcf0021f9d [term 2 FOLLOWER]: Leader election won for term 2
I20260926 07:33:24.266791 6249 raft_consensus.cc:686] T 00000000000000000000000000000000 P 788e7a47032d4f458d5493fcf0021f9d [term 2 LEADER]: Becoming Leader. State: Replica: 788e7a47032d4f458d5493fcf0021f9d, State: Running, Role: LEADER
I20260926 07:33:24.267186 6249 consensus_queue.cc:237] T 00000000000000000000000000000000 P 788e7a47032d4f458d5493fcf0021f9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788e7a47032d4f458d5493fcf0021f9d" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 32891 } }
I20260926 07:33:24.267872 6245 sys_catalog.cc:564] T 00000000000000000000000000000000 P 788e7a47032d4f458d5493fcf0021f9d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:33:24.268280 6249 sys_catalog.cc:455] T 00000000000000000000000000000000 P 788e7a47032d4f458d5493fcf0021f9d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 788e7a47032d4f458d5493fcf0021f9d. Latest consensus state: current_term: 2 leader_uuid: "788e7a47032d4f458d5493fcf0021f9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788e7a47032d4f458d5493fcf0021f9d" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 32891 } } }
I20260926 07:33:24.268440 6249 sys_catalog.cc:458] T 00000000000000000000000000000000 P 788e7a47032d4f458d5493fcf0021f9d [sys.catalog]: This master's current role is: LEADER
I20260926 07:33:24.268812 6250 sys_catalog.cc:455] T 00000000000000000000000000000000 P 788e7a47032d4f458d5493fcf0021f9d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "788e7a47032d4f458d5493fcf0021f9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788e7a47032d4f458d5493fcf0021f9d" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 32891 } } }
I20260926 07:33:24.268991 6250 sys_catalog.cc:458] T 00000000000000000000000000000000 P 788e7a47032d4f458d5493fcf0021f9d [sys.catalog]: This master's current role is: LEADER
I20260926 07:33:24.269311 6253 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:33:24.275098 6253 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:33:24.275926 6253 catalog_manager.cc:1259] Loaded cluster ID: 8716aa67f32a4bd89be30c51639b80e4
I20260926 07:33:24.276000 6253 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:33:24.278723 6253 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:33:24.279629 6253 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 788e7a47032d4f458d5493fcf0021f9d: Loaded TSK: 0
I20260926 07:33:24.298384 6253 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 788e7a47032d4f458d5493fcf0021f9d: Generated new TSK 1
I20260926 07:33:24.298725 6253 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 07:33:24.356900 6247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:33:24.357203 6247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:33:24.357251 6247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:33:24.361757 6247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:33:24.361855 6247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:33:24.361923 6247 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.53.193
I20260926 07:33:24.367862 6247 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:36403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.193.keytab
--principal=kudu/127.22.53.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.53.193:40583
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.53.193
--webserver_port=46431
--tserver_master_addrs=127.22.53.254:32891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6247
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.53.193
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:33:24.369297 6247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:33:24.370497 6247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:33:24.381512 6272 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:24.381531 6273 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:24.381531 6275 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:33:24.382095 6247 server_base.cc:1034] running on GCE node
I20260926 07:33:24.382570 6247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:33:24.383322 6247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:33:24.384697 6247 hybrid_clock.cc:648] HybridClock initialized: now 1790408004384657 us; error 40 us; skew 500 ppm
Sep 26 07:33:24 dist-test-slave-wdjc krb5kdc[5629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408004, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:33:24.387912 6247 init.cc:377] Logged in from keytab as kudu/127.22.53.193@KRBTEST.COM (short username kudu)
I20260926 07:33:24.389140 6247 webserver.cc:460] Webserver started at http://127.22.53.193:46431/ using document root <none> and password file <none>
I20260926 07:33:24.389822 6247 fs_manager.cc:359] Metadata directory not provided
I20260926 07:33:24.389902 6247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:33:24.394357 6247 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 07:33:24.396812 6282 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:24.398309 6247 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:33:24.398432 6247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-0/wal
uuid: "35ab6a8afa1a4639b35923c7d881a6c7"
format_stamp: "Formatted at 2026-09-26 07:33:03 on dist-test-slave-wdjc"
server_key: "e1ffe45e46c039841ee805ddd4d70b4d"
server_key_iv: "d0b5410a929a21d0fd39ce354f1d6709"
server_key_version: "encryptionkey@0"
I20260926 07:33:24.399065 6247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:33:24.410276 6247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:33:24.413903 6247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:33:24.414132 6247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:33:24.414870 6247 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:33:24.416173 6289 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 07:33:24.420725 6247 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 07:33:24.420814 6247 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260926 07:33:24.420895 6247 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 07:33:24.422910 6247 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 07:33:24.422993 6247 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260926 07:33:24.423282 6289 tablet_bootstrap.cc:492] T e514e01e6b1e4f0aabf9b2b562359829 P 35ab6a8afa1a4639b35923c7d881a6c7: Bootstrap starting.
I20260926 07:33:24.435148 6247 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.193:40583
I20260926 07:33:24.435477 6396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.193:40583 every 8 connection(s)
I20260926 07:33:24.436592 6247 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-0/data/info.pb
I20260926 07:33:24.440125 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 6247
I20260926 07:33:24.440678 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.53.194:40153
--local_ip_for_outbound_sockets=127.22.53.194
--tserver_master_addrs=127.22.53.254:32891
--webserver_port=36643
--webserver_interface=127.22.53.194
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.194.keytab
--principal=kudu/127.22.53.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:36403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:33:24.446518 6289 log.cc:824] T e514e01e6b1e4f0aabf9b2b562359829 P 35ab6a8afa1a4639b35923c7d881a6c7: Log is configured to *not* fsync() on all Append() calls
Sep 26 07:33:24 dist-test-slave-wdjc krb5kdc[5629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408004, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.193@KRBTEST.COM for kudu/127.22.53.254@KRBTEST.COM
I20260926 07:33:24.455391 6289 tablet_bootstrap.cc:492] T e514e01e6b1e4f0aabf9b2b562359829 P 35ab6a8afa1a4639b35923c7d881a6c7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 07:33:24.455883 6289 tablet_bootstrap.cc:492] T e514e01e6b1e4f0aabf9b2b562359829 P 35ab6a8afa1a4639b35923c7d881a6c7: Bootstrap complete.
I20260926 07:33:24.457109 6400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:24.438925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:32891 (local address 127.22.53.193:46623)
0926 07:33:24.440253 (+ 1328us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:24.440312 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 07:33:24.441598 (+ 1286us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:33:24.442932 (+ 1334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:33:24.442940 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:33:24.443382 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:33:24.443846 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:24.443862 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:24.445240 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:24.445244 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:24.446147 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:24.446158 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:24.447107 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:24.447112 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:24.447305 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:24.448003 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:33:24.448032 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:33:24.450124 (+ 2092us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:33:24.453265 (+ 3141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:33:24.453271 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:33:24.453284 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:33:24.453667 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:33:24.454162 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:33:24.454165 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:33:24.454167 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:33:24.454350 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:33:24.455096 (+ 746us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:33:24.455104 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:33:24.455402 (+ 298us) client_negotiation.cc:769] Sending connection context
0926 07:33:24.455606 (+ 204us) client_negotiation.cc:240] Negotiation successful
0926 07:33:24.455986 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1106,"thread_start_us":181,"threads_started":1}
I20260926 07:33:24.457345 6289 ts_tablet_manager.cc:1397] T e514e01e6b1e4f0aabf9b2b562359829 P 35ab6a8afa1a4639b35923c7d881a6c7: Time spent bootstrapping tablet: real 0.034s user 0.019s sys 0.014s
I20260926 07:33:24.458894 6399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:24.438875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.193:46623 (local address 127.22.53.254:32891)
0926 07:33:24.439304 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:24.439319 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:24.440464 (+ 1145us) server_negotiation.cc:407] Connection header received
0926 07:33:24.442158 (+ 1694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:24.442169 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:24.442500 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:24.442864 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:24.444046 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:24.445088 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:24.446307 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:24.446951 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:24.447149 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:24.450413 (+ 3264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:24.450422 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:24.450447 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:24.450498 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:24.453073 (+ 2575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:24.453824 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:24.453833 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:24.453840 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:24.454031 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:24.454463 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:24.454467 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:24.454469 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:24.454962 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:24.455101 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:24.457655 (+ 2554us) server_negotiation.cc:299] Negotiation successful
0926 07:33:24.458012 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":104,"threads_started":1}
I20260926 07:33:24.460390 6289 raft_consensus.cc:348] T e514e01e6b1e4f0aabf9b2b562359829 P 35ab6a8afa1a4639b35923c7d881a6c7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ab6a8afa1a4639b35923c7d881a6c7" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 40583 } } peers { permanent_uuid: "5a0860bc51e14488b4f1810c7b69e9bd" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 35847 } } peers { permanent_uuid: "ed471121813d4d1a9eb3f4e1c02da65b" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 40153 } }
I20260926 07:33:24.460923 6289 raft_consensus.cc:729] T e514e01e6b1e4f0aabf9b2b562359829 P 35ab6a8afa1a4639b35923c7d881a6c7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35ab6a8afa1a4639b35923c7d881a6c7, State: Initialized, Role: FOLLOWER
I20260926 07:33:24.461455 6397 heartbeater.cc:347] Connected to a master server at 127.22.53.254:32891
I20260926 07:33:24.461421 6289 consensus_queue.cc:260] T e514e01e6b1e4f0aabf9b2b562359829 P 35ab6a8afa1a4639b35923c7d881a6c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ab6a8afa1a4639b35923c7d881a6c7" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 40583 } } peers { permanent_uuid: "5a0860bc51e14488b4f1810c7b69e9bd" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 35847 } } peers { permanent_uuid: "ed471121813d4d1a9eb3f4e1c02da65b" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 40153 } }
I20260926 07:33:24.461714 6397 heartbeater.cc:464] Registering TS with master...
I20260926 07:33:24.462321 6397 heartbeater.cc:511] Master 127.22.53.254:32891 requested a full tablet report, sending...
I20260926 07:33:24.462448 6289 ts_tablet_manager.cc:1428] T e514e01e6b1e4f0aabf9b2b562359829 P 35ab6a8afa1a4639b35923c7d881a6c7: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260926 07:33:24.465328 6209 ts_manager.cc:194] Registered new tserver with Master: 35ab6a8afa1a4639b35923c7d881a6c7 (127.22.53.193:40583)
I20260926 07:33:24.469108 6209 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.53.193@KRBTEST.COM'} at 127.22.53.193:46623
I20260926 07:33:24.472214 6397 heartbeater.cc:503] Master 127.22.53.254:32891 was elected leader, sending a full tablet report...
Sep 26 07:33:24 dist-test-slave-wdjc krb5kdc[5629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408004, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.254@KRBTEST.COM for kudu/127.22.53.193@KRBTEST.COM
I20260926 07:33:24.482721 6405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:24.468730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50912 (local address 127.22.53.193:40583)
0926 07:33:24.469216 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:24.469299 (+ 83us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:24.469699 (+ 400us) server_negotiation.cc:407] Connection header received
0926 07:33:24.469914 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:24.469921 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:24.470179 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:24.470348 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:24.473319 (+ 2971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:24.474436 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:24.475332 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:24.475823 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:24.475936 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:24.478504 (+ 2568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:24.478515 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:24.478526 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:24.478553 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:24.480411 (+ 1858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:24.481167 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:24.481172 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:24.481174 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:24.481263 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:24.481580 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:24.481582 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:24.481584 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:24.481884 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:24.481992 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:24.482422 (+ 430us) server_negotiation.cc:299] Negotiation successful
0926 07:33:24.482580 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":128,"threads_started":1}
I20260926 07:33:24.482749 6404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:24.468568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.193:40583 (local address 127.0.0.1:50912)
0926 07:33:24.469537 (+ 969us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:24.469583 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 07:33:24.469750 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:33:24.470358 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:33:24.470364 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:33:24.472980 (+ 2616us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:33:24.473160 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:24.473170 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:24.474560 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:24.474562 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:24.475197 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:24.475207 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:24.475947 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:24.475949 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:24.476035 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:24.476733 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:33:24.476746 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:33:24.478315 (+ 1569us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:33:24.480593 (+ 2278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:33:24.480610 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:33:24.480617 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:33:24.481016 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:33:24.481369 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:33:24.481372 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:33:24.481374 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:33:24.481478 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:33:24.482007 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:33:24.482014 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:33:24.482176 (+ 162us) client_negotiation.cc:769] Sending connection context
0926 07:33:24.482328 (+ 152us) client_negotiation.cc:240] Negotiation successful
0926 07:33:24.482569 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":789,"thread_start_us":119,"threads_started":1}
I20260926 07:33:24.483903 6331 tablet_service.cc:1514] Processing DeleteTablet for tablet e514e01e6b1e4f0aabf9b2b562359829 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:33:03 UTC) from {username='kudu', principal='kudu/127.22.53.254@KRBTEST.COM'} at 127.0.0.1:50912
I20260926 07:33:24.484999 6406 tablet_replica.cc:330] stopping tablet replica
I20260926 07:33:24.485369 6406 raft_consensus.cc:2229] T e514e01e6b1e4f0aabf9b2b562359829 P 35ab6a8afa1a4639b35923c7d881a6c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:33:24.485648 6406 raft_consensus.cc:2258] T e514e01e6b1e4f0aabf9b2b562359829 P 35ab6a8afa1a4639b35923c7d881a6c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:33:24.488297 6406 ts_tablet_manager.cc:1905] T e514e01e6b1e4f0aabf9b2b562359829 P 35ab6a8afa1a4639b35923c7d881a6c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:33:24.493656 6406 ts_tablet_manager.cc:1918] T e514e01e6b1e4f0aabf9b2b562359829 P 35ab6a8afa1a4639b35923c7d881a6c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:33:24.493810 6406 log.cc:1196] T e514e01e6b1e4f0aabf9b2b562359829 P 35ab6a8afa1a4639b35923c7d881a6c7: Deleting WAL directory at /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-0/wal/wals/e514e01e6b1e4f0aabf9b2b562359829
I20260926 07:33:24.494102 6406 ts_tablet_manager.cc:1939] T e514e01e6b1e4f0aabf9b2b562359829 P 35ab6a8afa1a4639b35923c7d881a6c7: Deleting consensus metadata
I20260926 07:33:24.496383 6194 catalog_manager.cc:4841] TS 35ab6a8afa1a4639b35923c7d881a6c7 (127.22.53.193:40583): tablet e514e01e6b1e4f0aabf9b2b562359829 (table test-table [id=2194e33bd08a4ad88b469277d6318869]) successfully deleted
W20260926 07:33:24.574115 6401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:33:24.574434 6401 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:33:24.574481 6401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:33:24.578778 6401 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:33:24.578900 6401 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:33:24.579020 6401 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.53.194
I20260926 07:33:24.584645 6401 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:36403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.194.keytab
--principal=kudu/127.22.53.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.53.194:40153
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.53.194
--webserver_port=36643
--tserver_master_addrs=127.22.53.254:32891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6401
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.53.194
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:33:24.585772 6401 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:33:24.586792 6401 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:33:24.597656 6411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:24.597666 6412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:24.597656 6414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:33:24.597986 6401 server_base.cc:1034] running on GCE node
I20260926 07:33:24.598464 6401 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:33:24.599203 6401 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:33:24.600436 6401 hybrid_clock.cc:648] HybridClock initialized: now 1790408004600396 us; error 58 us; skew 500 ppm
Sep 26 07:33:24 dist-test-slave-wdjc krb5kdc[5629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408004, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:33:24.604027 6401 init.cc:377] Logged in from keytab as kudu/127.22.53.194@KRBTEST.COM (short username kudu)
I20260926 07:33:24.605381 6401 webserver.cc:460] Webserver started at http://127.22.53.194:36643/ using document root <none> and password file <none>
I20260926 07:33:24.606019 6401 fs_manager.cc:359] Metadata directory not provided
I20260926 07:33:24.606112 6401 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:33:24.611006 6401 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 07:33:24.613649 6421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:24.615157 6401 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:33:24.615307 6401 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-1/wal
uuid: "ed471121813d4d1a9eb3f4e1c02da65b"
format_stamp: "Formatted at 2026-09-26 07:33:03 on dist-test-slave-wdjc"
server_key: "106f575b8ff401addcc90eff49306219"
server_key_iv: "ba0883ae30d30c700a88b832a8a47804"
server_key_version: "encryptionkey@0"
I20260926 07:33:24.615914 6401 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:33:24.627158 6401 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:33:24.630925 6401 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:33:24.631172 6401 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:33:24.631857 6401 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:33:24.632784 6401 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:33:24.632838 6401 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:24.632879 6401 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:33:24.632894 6401 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:24.642714 6401 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.194:40153
I20260926 07:33:24.642761 6534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.194:40153 every 8 connection(s)
I20260926 07:33:24.643831 6401 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-1/data/info.pb
I20260926 07:33:24.650401 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 6401
I20260926 07:33:24.650954 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.53.195:35847
--local_ip_for_outbound_sockets=127.22.53.195
--tserver_master_addrs=127.22.53.254:32891
--webserver_port=43537
--webserver_interface=127.22.53.195
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.195.keytab
--principal=kudu/127.22.53.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:36403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 07:33:24 dist-test-slave-wdjc krb5kdc[5629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408004, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.194@KRBTEST.COM for kudu/127.22.53.254@KRBTEST.COM
I20260926 07:33:24.661526 6399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:24.645931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.194:32843 (local address 127.22.53.254:32891)
0926 07:33:24.646096 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:24.646100 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:24.647061 (+ 961us) server_negotiation.cc:407] Connection header received
0926 07:33:24.648486 (+ 1425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:24.648489 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:24.648608 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:24.648703 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:24.650144 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:24.650757 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:24.652890 (+ 2133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:24.653354 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:24.653509 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:24.656693 (+ 3184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:24.656699 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:24.656704 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:24.656748 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:24.658711 (+ 1963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:24.659431 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:24.659438 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:24.659441 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:24.659551 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:24.659950 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:24.659955 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:24.659958 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:24.660254 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:24.660391 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:24.661070 (+ 679us) server_negotiation.cc:299] Negotiation successful
0926 07:33:24.661274 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 07:33:24.662321 6537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:24.646353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:32891 (local address 127.22.53.194:32843)
0926 07:33:24.646882 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:24.646930 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 07:33:24.648252 (+ 1322us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:33:24.648907 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:33:24.648914 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:33:24.649378 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:33:24.649921 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:24.649937 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:24.650943 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:24.650947 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:24.652704 (+ 1757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:24.652729 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:24.653570 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:24.653574 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:24.653744 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:24.654475 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:33:24.654496 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:33:24.656427 (+ 1931us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:33:24.658886 (+ 2459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:33:24.658893 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:33:24.658909 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:33:24.659301 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:33:24.659668 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:33:24.659671 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:33:24.659672 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:33:24.659835 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:33:24.660431 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:33:24.660437 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:33:24.660793 (+ 356us) client_negotiation.cc:769] Sending connection context
0926 07:33:24.660985 (+ 192us) client_negotiation.cc:240] Negotiation successful
0926 07:33:24.661304 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":127,"threads_started":1}
I20260926 07:33:24.663774 6535 heartbeater.cc:347] Connected to a master server at 127.22.53.254:32891
I20260926 07:33:24.664037 6535 heartbeater.cc:464] Registering TS with master...
I20260926 07:33:24.664669 6535 heartbeater.cc:511] Master 127.22.53.254:32891 requested a full tablet report, sending...
I20260926 07:33:24.665889 6209 ts_manager.cc:194] Registered new tserver with Master: ed471121813d4d1a9eb3f4e1c02da65b (127.22.53.194:40153)
I20260926 07:33:24.666513 6209 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.53.194@KRBTEST.COM'} at 127.22.53.194:32843
W20260926 07:33:24.776736 6538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:33:24.777083 6538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:33:24.777179 6538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:33:24.781913 6538 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:33:24.782001 6538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:33:24.782114 6538 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.53.195
I20260926 07:33:24.788797 6538 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:36403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.195.keytab
--principal=kudu/127.22.53.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.53.195:35847
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.53.195
--webserver_port=43537
--tserver_master_addrs=127.22.53.254:32891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6538
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.53.195
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:33:24.790239 6538 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:33:24.791417 6538 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:33:24.802505 6544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:24.802505 6546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:24.802505 6543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:33:24.802565 6538 server_base.cc:1034] running on GCE node
I20260926 07:33:24.803393 6538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:33:24.804121 6538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:33:24.805379 6538 hybrid_clock.cc:648] HybridClock initialized: now 1790408004805323 us; error 72 us; skew 500 ppm
Sep 26 07:33:24 dist-test-slave-wdjc krb5kdc[5629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408004, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:33:24.808431 6538 init.cc:377] Logged in from keytab as kudu/127.22.53.195@KRBTEST.COM (short username kudu)
I20260926 07:33:24.809484 6538 webserver.cc:460] Webserver started at http://127.22.53.195:43537/ using document root <none> and password file <none>
I20260926 07:33:24.810122 6538 fs_manager.cc:359] Metadata directory not provided
I20260926 07:33:24.810248 6538 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:33:24.814714 6538 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 07:33:24.817268 6553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:24.818756 6538 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:33:24.818925 6538 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-2/wal
uuid: "5a0860bc51e14488b4f1810c7b69e9bd"
format_stamp: "Formatted at 2026-09-26 07:33:03 on dist-test-slave-wdjc"
server_key: "38343cbcbaa2c117f57ac9160cdf2edf"
server_key_iv: "9b0dc03585259ea20641300e57bba641"
server_key_version: "encryptionkey@0"
I20260926 07:33:24.819619 6538 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:33:24.836058 6538 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:33:24.839965 6538 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:33:24.840211 6538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:33:24.840966 6538 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:33:24.841948 6538 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:33:24.842028 6538 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:24.842105 6538 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:33:24.842154 6538 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:24.851804 6538 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.195:35847
I20260926 07:33:24.851862 6666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.195:35847 every 8 connection(s)
I20260926 07:33:24.853035 6538 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-2/data/info.pb
I20260926 07:33:24.859505 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 6538
Sep 26 07:33:24 dist-test-slave-wdjc krb5kdc[5629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408004, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.195@KRBTEST.COM for kudu/127.22.53.254@KRBTEST.COM
I20260926 07:33:24.870147 6671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:24.860402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45698 (local address 127.22.53.254:32891)
0926 07:33:24.860718 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:24.860721 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:24.860917 (+ 196us) server_negotiation.cc:407] Connection header received
0926 07:33:24.860970 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:24.860972 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:24.861019 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:24.861147 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:24.862287 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:24.863089 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:24.864343 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:24.864701 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:24.864872 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:24.866248 (+ 1376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:24.866256 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:24.866262 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:24.866317 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:24.868237 (+ 1920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:24.868795 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:24.868801 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:24.868806 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:24.868893 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:24.869184 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:24.869187 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:24.869189 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:24.869419 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:24.869653 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:24.869793 (+ 140us) server_negotiation.cc:299] Negotiation successful
0926 07:33:24.869928 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":63,"threads_started":1}
I20260926 07:33:24.871284 6399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:24.855276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.195:57163 (local address 127.22.53.254:32891)
0926 07:33:24.855435 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:24.855440 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:24.856395 (+ 955us) server_negotiation.cc:407] Connection header received
0926 07:33:24.857894 (+ 1499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:24.857897 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:24.857954 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:24.858067 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:24.860452 (+ 2385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:24.861383 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:24.863423 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:24.863817 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:24.863929 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:24.866968 (+ 3039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:24.866971 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:24.866973 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:24.867007 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:24.868772 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:24.869420 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:24.869425 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:24.869427 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:24.869544 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:24.869911 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:24.869916 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:24.869918 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:24.870207 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:24.870307 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:24.871023 (+ 716us) server_negotiation.cc:299] Negotiation successful
0926 07:33:24.871143 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 07:33:24.872457 6669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:24.855604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:32891 (local address 127.22.53.195:57163)
0926 07:33:24.856191 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:24.856241 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 07:33:24.857615 (+ 1374us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:33:24.858516 (+ 901us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:33:24.858530 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:33:24.859113 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:33:24.860214 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:24.860256 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:24.861607 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:24.861610 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:24.863264 (+ 1654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:24.863301 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:24.863983 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:24.863986 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:24.864153 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:24.864989 (+ 836us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:33:24.865007 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:33:24.866764 (+ 1757us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:33:24.868897 (+ 2133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:33:24.868904 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:33:24.868925 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:33:24.869295 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:33:24.869648 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:33:24.869651 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:33:24.869653 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:33:24.869810 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:33:24.870364 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:33:24.870371 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:33:24.870744 (+ 373us) client_negotiation.cc:769] Sending connection context
0926 07:33:24.870985 (+ 241us) client_negotiation.cc:240] Negotiation successful
0926 07:33:24.871384 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":138,"threads_started":1}
I20260926 07:33:24.873848 6667 heartbeater.cc:347] Connected to a master server at 127.22.53.254:32891
I20260926 07:33:24.874110 6667 heartbeater.cc:464] Registering TS with master...
I20260926 07:33:24.874629 6667 heartbeater.cc:511] Master 127.22.53.254:32891 requested a full tablet report, sending...
I20260926 07:33:24.876024 6209 ts_manager.cc:194] Registered new tserver with Master: 5a0860bc51e14488b4f1810c7b69e9bd (127.22.53.195:35847)
I20260926 07:33:24.876924 6209 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.53.195@KRBTEST.COM'} at 127.22.53.195:57163
I20260926 07:33:24.884488 22743 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:33:24.890192 6399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:24.885381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45710 (local address 127.22.53.254:32891)
0926 07:33:24.885541 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:24.885555 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:24.885845 (+ 290us) server_negotiation.cc:407] Connection header received
0926 07:33:24.886121 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:24.886125 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:24.886278 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:24.886396 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:33:24.886730 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:24.887368 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:24.888708 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:24.888972 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:24.889092 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:24.889525 (+ 433us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:33:24.889713 (+ 188us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0926 07:33:24.890043 (+ 330us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45710: token expired
Metrics: {"server-negotiator.queue_time_us":61}
W20260926 07:33:24.890300 6399 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45710: token expired
W20260926 07:33:24.891068 6137 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.22.53.254:32891)
I20260926 07:33:24.901598 6399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:24.891681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45718 (local address 127.22.53.254:32891)
0926 07:33:24.891839 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:24.891844 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:24.891917 (+ 73us) server_negotiation.cc:407] Connection header received
0926 07:33:24.892095 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:24.892099 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:24.892151 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:24.892273 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:24.893095 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:24.893640 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:24.894549 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:24.894771 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:24.894882 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:24.896939 (+ 2057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:24.896960 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:24.896966 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:24.897003 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:24.899208 (+ 2205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:24.900053 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:24.900057 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:24.900059 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:24.900226 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:24.900588 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:24.900592 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:24.900594 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:24.900820 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:24.900991 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:24.901262 (+ 271us) server_negotiation.cc:299] Negotiation successful
0926 07:33:24.901403 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 07:33:24.905880 6209 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45718:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:33:24.910653 6209 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:33:24.920631 6331 tablet_service.cc:1467] Processing CreateTablet for tablet 561a92b62f5e4870a8281e704ae2e4fa (DEFAULT_TABLE table=test-table [id=791126f3a5ad467a83e3c553b200a4b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:33:24.921428 6331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 561a92b62f5e4870a8281e704ae2e4fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:33:24.924752 6289 tablet_bootstrap.cc:492] T 561a92b62f5e4870a8281e704ae2e4fa P 35ab6a8afa1a4639b35923c7d881a6c7: Bootstrap starting.
I20260926 07:33:24.925770 6289 tablet_bootstrap.cc:654] T 561a92b62f5e4870a8281e704ae2e4fa P 35ab6a8afa1a4639b35923c7d881a6c7: Neither blocks nor log segments found. Creating new log.
I20260926 07:33:24.926247 6678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:24.920238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59148 (local address 127.22.53.194:40153)
0926 07:33:24.920643 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:24.920654 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:24.921391 (+ 737us) server_negotiation.cc:407] Connection header received
0926 07:33:24.921496 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:24.921503 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:24.921703 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:24.921881 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:33:24.922358 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:24.923225 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:24.924563 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:24.925488 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:24.925623 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:24.925811 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:24.925985 (+ 174us) server_negotiation.cc:299] Negotiation successful
0926 07:33:24.926093 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":167,"threads_started":1}
I20260926 07:33:24.926276 6404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:24.919434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.195:35847 (local address 127.0.0.1:54646)
0926 07:33:24.919626 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:24.919642 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:33:24.919731 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:33:24.920927 (+ 1196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:33:24.920931 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:33:24.920951 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:33:24.921078 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:24.921089 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:24.922908 (+ 1819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:24.922915 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:24.924221 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:24.924234 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:24.925747 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:24.925752 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:24.925887 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:24.925914 (+ 27us) client_negotiation.cc:769] Sending connection context
0926 07:33:24.925978 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 07:33:24.926073 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"mutex_wait_us":2}
I20260926 07:33:24.926169 6677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:24.919783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.194:40153 (local address 127.0.0.1:59148)
0926 07:33:24.921163 (+ 1380us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:24.921181 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 07:33:24.921285 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:33:24.922087 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:33:24.922091 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:33:24.922108 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:33:24.922223 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:24.922230 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:24.923338 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:24.923343 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:24.924395 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:24.924408 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:24.925636 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:24.925639 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:24.925742 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:24.925835 (+ 93us) client_negotiation.cc:769] Sending connection context
0926 07:33:24.925907 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 07:33:24.926005 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1271,"mutex_wait_us":54,"thread_start_us":139,"threads_started":1}
I20260926 07:33:24.927585 6289 tablet_bootstrap.cc:492] T 561a92b62f5e4870a8281e704ae2e4fa P 35ab6a8afa1a4639b35923c7d881a6c7: No bootstrap required, opened a new log
I20260926 07:33:24.927711 6676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:24.919658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54646 (local address 127.22.53.195:35847)
0926 07:33:24.920299 (+ 641us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:24.920309 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:24.920331 (+ 22us) server_negotiation.cc:407] Connection header received
0926 07:33:24.920407 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:24.920412 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:24.920732 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:24.920925 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:33:24.921266 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:24.922760 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:24.924401 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:24.925592 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:24.926277 (+ 685us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:24.926432 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:24.926489 (+ 57us) server_negotiation.cc:299] Negotiation successful
0926 07:33:24.926602 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":524,"thread_start_us":144,"threads_started":1}
I20260926 07:33:24.927719 6289 ts_tablet_manager.cc:1397] T 561a92b62f5e4870a8281e704ae2e4fa P 35ab6a8afa1a4639b35923c7d881a6c7: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260926 07:33:24.928278 6289 raft_consensus.cc:348] T 561a92b62f5e4870a8281e704ae2e4fa P 35ab6a8afa1a4639b35923c7d881a6c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a0860bc51e14488b4f1810c7b69e9bd" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 35847 } } peers { permanent_uuid: "ed471121813d4d1a9eb3f4e1c02da65b" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 40153 } } peers { permanent_uuid: "35ab6a8afa1a4639b35923c7d881a6c7" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 40583 } }
I20260926 07:33:24.928422 6289 raft_consensus.cc:374] T 561a92b62f5e4870a8281e704ae2e4fa P 35ab6a8afa1a4639b35923c7d881a6c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:33:24.928471 6289 raft_consensus.cc:729] T 561a92b62f5e4870a8281e704ae2e4fa P 35ab6a8afa1a4639b35923c7d881a6c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35ab6a8afa1a4639b35923c7d881a6c7, State: Initialized, Role: FOLLOWER
I20260926 07:33:24.928638 6289 consensus_queue.cc:260] T 561a92b62f5e4870a8281e704ae2e4fa P 35ab6a8afa1a4639b35923c7d881a6c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a0860bc51e14488b4f1810c7b69e9bd" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 35847 } } peers { permanent_uuid: "ed471121813d4d1a9eb3f4e1c02da65b" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 40153 } } peers { permanent_uuid: "35ab6a8afa1a4639b35923c7d881a6c7" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 40583 } }
I20260926 07:33:24.929068 6289 ts_tablet_manager.cc:1428] T 561a92b62f5e4870a8281e704ae2e4fa P 35ab6a8afa1a4639b35923c7d881a6c7: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260926 07:33:24.930217 6469 tablet_service.cc:1467] Processing CreateTablet for tablet 561a92b62f5e4870a8281e704ae2e4fa (DEFAULT_TABLE table=test-table [id=791126f3a5ad467a83e3c553b200a4b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:33:24.931226 6601 tablet_service.cc:1467] Processing CreateTablet for tablet 561a92b62f5e4870a8281e704ae2e4fa (DEFAULT_TABLE table=test-table [id=791126f3a5ad467a83e3c553b200a4b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:33:24.931561 6469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 561a92b62f5e4870a8281e704ae2e4fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:33:24.932123 6601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 561a92b62f5e4870a8281e704ae2e4fa. 1 dirs total, 0 dirs full, 0 dirs failed
W20260926 07:33:24.938081 6398 tablet.cc:2369] T 561a92b62f5e4870a8281e704ae2e4fa P 35ab6a8afa1a4639b35923c7d881a6c7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:33:24.941211 6680 tablet_bootstrap.cc:492] T 561a92b62f5e4870a8281e704ae2e4fa P ed471121813d4d1a9eb3f4e1c02da65b: Bootstrap starting.
I20260926 07:33:24.942252 6681 tablet_bootstrap.cc:492] T 561a92b62f5e4870a8281e704ae2e4fa P 5a0860bc51e14488b4f1810c7b69e9bd: Bootstrap starting.
I20260926 07:33:24.943979 6680 tablet_bootstrap.cc:654] T 561a92b62f5e4870a8281e704ae2e4fa P ed471121813d4d1a9eb3f4e1c02da65b: Neither blocks nor log segments found. Creating new log.
I20260926 07:33:24.944144 6681 tablet_bootstrap.cc:654] T 561a92b62f5e4870a8281e704ae2e4fa P 5a0860bc51e14488b4f1810c7b69e9bd: Neither blocks nor log segments found. Creating new log.
I20260926 07:33:24.945036 6680 log.cc:824] T 561a92b62f5e4870a8281e704ae2e4fa P ed471121813d4d1a9eb3f4e1c02da65b: Log is configured to *not* fsync() on all Append() calls
I20260926 07:33:24.945466 6681 log.cc:824] T 561a92b62f5e4870a8281e704ae2e4fa P 5a0860bc51e14488b4f1810c7b69e9bd: Log is configured to *not* fsync() on all Append() calls
I20260926 07:33:24.947679 6681 tablet_bootstrap.cc:492] T 561a92b62f5e4870a8281e704ae2e4fa P 5a0860bc51e14488b4f1810c7b69e9bd: No bootstrap required, opened a new log
I20260926 07:33:24.947679 6680 tablet_bootstrap.cc:492] T 561a92b62f5e4870a8281e704ae2e4fa P ed471121813d4d1a9eb3f4e1c02da65b: No bootstrap required, opened a new log
I20260926 07:33:24.947921 6680 ts_tablet_manager.cc:1397] T 561a92b62f5e4870a8281e704ae2e4fa P ed471121813d4d1a9eb3f4e1c02da65b: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260926 07:33:24.947921 6681 ts_tablet_manager.cc:1397] T 561a92b62f5e4870a8281e704ae2e4fa P 5a0860bc51e14488b4f1810c7b69e9bd: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260926 07:33:24.952311 6680 raft_consensus.cc:348] T 561a92b62f5e4870a8281e704ae2e4fa P ed471121813d4d1a9eb3f4e1c02da65b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a0860bc51e14488b4f1810c7b69e9bd" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 35847 } } peers { permanent_uuid: "ed471121813d4d1a9eb3f4e1c02da65b" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 40153 } } peers { permanent_uuid: "35ab6a8afa1a4639b35923c7d881a6c7" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 40583 } }
I20260926 07:33:24.952311 6681 raft_consensus.cc:348] T 561a92b62f5e4870a8281e704ae2e4fa P 5a0860bc51e14488b4f1810c7b69e9bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a0860bc51e14488b4f1810c7b69e9bd" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 35847 } } peers { permanent_uuid: "ed471121813d4d1a9eb3f4e1c02da65b" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 40153 } } peers { permanent_uuid: "35ab6a8afa1a4639b35923c7d881a6c7" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 40583 } }
I20260926 07:33:24.952718 6680 raft_consensus.cc:374] T 561a92b62f5e4870a8281e704ae2e4fa P ed471121813d4d1a9eb3f4e1c02da65b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:33:24.952718 6681 raft_consensus.cc:374] T 561a92b62f5e4870a8281e704ae2e4fa P 5a0860bc51e14488b4f1810c7b69e9bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:33:24.952786 6681 raft_consensus.cc:729] T 561a92b62f5e4870a8281e704ae2e4fa P 5a0860bc51e14488b4f1810c7b69e9bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a0860bc51e14488b4f1810c7b69e9bd, State: Initialized, Role: FOLLOWER
I20260926 07:33:24.952785 6680 raft_consensus.cc:729] T 561a92b62f5e4870a8281e704ae2e4fa P ed471121813d4d1a9eb3f4e1c02da65b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed471121813d4d1a9eb3f4e1c02da65b, State: Initialized, Role: FOLLOWER
I20260926 07:33:24.953330 6681 consensus_queue.cc:260] T 561a92b62f5e4870a8281e704ae2e4fa P 5a0860bc51e14488b4f1810c7b69e9bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a0860bc51e14488b4f1810c7b69e9bd" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 35847 } } peers { permanent_uuid: "ed471121813d4d1a9eb3f4e1c02da65b" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 40153 } } peers { permanent_uuid: "35ab6a8afa1a4639b35923c7d881a6c7" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 40583 } }
I20260926 07:33:24.953330 6680 consensus_queue.cc:260] T 561a92b62f5e4870a8281e704ae2e4fa P ed471121813d4d1a9eb3f4e1c02da65b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a0860bc51e14488b4f1810c7b69e9bd" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 35847 } } peers { permanent_uuid: "ed471121813d4d1a9eb3f4e1c02da65b" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 40153 } } peers { permanent_uuid: "35ab6a8afa1a4639b35923c7d881a6c7" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 40583 } }
I20260926 07:33:24.954139 6667 heartbeater.cc:503] Master 127.22.53.254:32891 was elected leader, sending a full tablet report...
I20260926 07:33:24.954559 6680 ts_tablet_manager.cc:1428] T 561a92b62f5e4870a8281e704ae2e4fa P ed471121813d4d1a9eb3f4e1c02da65b: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260926 07:33:24.954559 6681 ts_tablet_manager.cc:1428] T 561a92b62f5e4870a8281e704ae2e4fa P 5a0860bc51e14488b4f1810c7b69e9bd: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260926 07:33:24.954653 6535 heartbeater.cc:503] Master 127.22.53.254:32891 was elected leader, sending a full tablet report...
I20260926 07:33:25.056653 6403 raft_consensus.cc:482] T 561a92b62f5e4870a8281e704ae2e4fa P 35ab6a8afa1a4639b35923c7d881a6c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:33:25.057036 6403 raft_consensus.cc:504] T 561a92b62f5e4870a8281e704ae2e4fa P 35ab6a8afa1a4639b35923c7d881a6c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a0860bc51e14488b4f1810c7b69e9bd" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 35847 } } peers { permanent_uuid: "ed471121813d4d1a9eb3f4e1c02da65b" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 40153 } } peers { permanent_uuid: "35ab6a8afa1a4639b35923c7d881a6c7" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 40583 } }
I20260926 07:33:25.058940 6403 leader_election.cc:288] T 561a92b62f5e4870a8281e704ae2e4fa P 35ab6a8afa1a4639b35923c7d881a6c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a0860bc51e14488b4f1810c7b69e9bd (127.22.53.195:35847), ed471121813d4d1a9eb3f4e1c02da65b (127.22.53.194:40153)
I20260926 07:33:25.063251 6676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:25.059309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.193:44373 (local address 127.22.53.195:35847)
0926 07:33:25.059475 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:25.059479 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:25.059843 (+ 364us) server_negotiation.cc:407] Connection header received
0926 07:33:25.060019 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:25.060027 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:25.060067 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:25.060154 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:33:25.060464 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:25.061040 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:25.062163 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:25.062587 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:25.062724 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:25.062782 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:25.063038 (+ 256us) server_negotiation.cc:299] Negotiation successful
0926 07:33:25.063116 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 07:33:25.063269 6687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:25.059208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.195:35847 (local address 127.22.53.193:44373)
0926 07:33:25.059731 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:25.059743 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 07:33:25.059891 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:33:25.060176 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:33:25.060179 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:33:25.060199 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:33:25.060343 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:25.060350 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:25.061160 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:25.061165 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:25.062024 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:25.062034 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:25.062724 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:25.062728 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:25.062872 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:25.062916 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 07:33:25.063007 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 07:33:25.063101 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"mutex_wait_us":44,"thread_start_us":87,"threads_started":1}
I20260926 07:33:25.063377 6678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:25.059450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.193:57895 (local address 127.22.53.194:40153)
0926 07:33:25.059586 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:25.059591 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:25.059748 (+ 157us) server_negotiation.cc:407] Connection header received
0926 07:33:25.059988 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:25.059994 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:25.060049 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:25.060151 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:33:25.060486 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:25.061061 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:25.062219 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:25.062669 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:25.062816 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:25.062881 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:25.063135 (+ 254us) server_negotiation.cc:299] Negotiation successful
0926 07:33:25.063218 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 07:33:25.063436 6686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:25.059202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.194:40153 (local address 127.22.53.193:57895)
0926 07:33:25.059618 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:25.059635 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 07:33:25.059835 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:33:25.060166 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:33:25.060169 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:33:25.060188 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:33:25.060350 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:25.060356 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:25.061209 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:25.061216 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:25.062070 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:25.062080 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:25.062816 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:25.062820 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:25.062984 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:25.063004 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 07:33:25.063100 (+ 96us) client_negotiation.cc:240] Negotiation successful
0926 07:33:25.063184 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"mutex_wait_us":52,"spinlock_wait_cycles":25472,"thread_start_us":117,"threads_started":1}
I20260926 07:33:25.064119 6621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "561a92b62f5e4870a8281e704ae2e4fa" candidate_uuid: "35ab6a8afa1a4639b35923c7d881a6c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a0860bc51e14488b4f1810c7b69e9bd" is_pre_election: true
I20260926 07:33:25.064344 6489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "561a92b62f5e4870a8281e704ae2e4fa" candidate_uuid: "35ab6a8afa1a4639b35923c7d881a6c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed471121813d4d1a9eb3f4e1c02da65b" is_pre_election: true
I20260926 07:33:25.064509 6621 raft_consensus.cc:2454] T 561a92b62f5e4870a8281e704ae2e4fa P 5a0860bc51e14488b4f1810c7b69e9bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35ab6a8afa1a4639b35923c7d881a6c7 in term 0.
I20260926 07:33:25.064622 6489 raft_consensus.cc:2454] T 561a92b62f5e4870a8281e704ae2e4fa P ed471121813d4d1a9eb3f4e1c02da65b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35ab6a8afa1a4639b35923c7d881a6c7 in term 0.
I20260926 07:33:25.065104 6283 leader_election.cc:302] T 561a92b62f5e4870a8281e704ae2e4fa P 35ab6a8afa1a4639b35923c7d881a6c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35ab6a8afa1a4639b35923c7d881a6c7, ed471121813d4d1a9eb3f4e1c02da65b; no voters:
I20260926 07:33:25.065387 6403 raft_consensus.cc:2781] T 561a92b62f5e4870a8281e704ae2e4fa P 35ab6a8afa1a4639b35923c7d881a6c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:33:25.065480 6403 raft_consensus.cc:482] T 561a92b62f5e4870a8281e704ae2e4fa P 35ab6a8afa1a4639b35923c7d881a6c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:33:25.065526 6403 raft_consensus.cc:3037] T 561a92b62f5e4870a8281e704ae2e4fa P 35ab6a8afa1a4639b35923c7d881a6c7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:33:25.066638 6403 raft_consensus.cc:504] T 561a92b62f5e4870a8281e704ae2e4fa P 35ab6a8afa1a4639b35923c7d881a6c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a0860bc51e14488b4f1810c7b69e9bd" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 35847 } } peers { permanent_uuid: "ed471121813d4d1a9eb3f4e1c02da65b" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 40153 } } peers { permanent_uuid: "35ab6a8afa1a4639b35923c7d881a6c7" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 40583 } }
I20260926 07:33:25.067050 6403 leader_election.cc:288] T 561a92b62f5e4870a8281e704ae2e4fa P 35ab6a8afa1a4639b35923c7d881a6c7 [CANDIDATE]: Term 1 election: Requested vote from peers 5a0860bc51e14488b4f1810c7b69e9bd (127.22.53.195:35847), ed471121813d4d1a9eb3f4e1c02da65b (127.22.53.194:40153)
I20260926 07:33:25.067485 6621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "561a92b62f5e4870a8281e704ae2e4fa" candidate_uuid: "35ab6a8afa1a4639b35923c7d881a6c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a0860bc51e14488b4f1810c7b69e9bd"
I20260926 07:33:25.067569 6489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "561a92b62f5e4870a8281e704ae2e4fa" candidate_uuid: "35ab6a8afa1a4639b35923c7d881a6c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed471121813d4d1a9eb3f4e1c02da65b"
I20260926 07:33:25.067667 6621 raft_consensus.cc:3037] T 561a92b62f5e4870a8281e704ae2e4fa P 5a0860bc51e14488b4f1810c7b69e9bd [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:33:25.067691 6489 raft_consensus.cc:3037] T 561a92b62f5e4870a8281e704ae2e4fa P ed471121813d4d1a9eb3f4e1c02da65b [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:33:25.069137 6489 raft_consensus.cc:2454] T 561a92b62f5e4870a8281e704ae2e4fa P ed471121813d4d1a9eb3f4e1c02da65b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35ab6a8afa1a4639b35923c7d881a6c7 in term 1.
I20260926 07:33:25.069137 6621 raft_consensus.cc:2454] T 561a92b62f5e4870a8281e704ae2e4fa P 5a0860bc51e14488b4f1810c7b69e9bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35ab6a8afa1a4639b35923c7d881a6c7 in term 1.
I20260926 07:33:25.069578 6283 leader_election.cc:302] T 561a92b62f5e4870a8281e704ae2e4fa P 35ab6a8afa1a4639b35923c7d881a6c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35ab6a8afa1a4639b35923c7d881a6c7, ed471121813d4d1a9eb3f4e1c02da65b; no voters:
I20260926 07:33:25.069821 6403 raft_consensus.cc:2781] T 561a92b62f5e4870a8281e704ae2e4fa P 35ab6a8afa1a4639b35923c7d881a6c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:33:25.070050 6403 raft_consensus.cc:686] T 561a92b62f5e4870a8281e704ae2e4fa P 35ab6a8afa1a4639b35923c7d881a6c7 [term 1 LEADER]: Becoming Leader. State: Replica: 35ab6a8afa1a4639b35923c7d881a6c7, State: Running, Role: LEADER
I20260926 07:33:25.070349 6403 consensus_queue.cc:237] T 561a92b62f5e4870a8281e704ae2e4fa P 35ab6a8afa1a4639b35923c7d881a6c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a0860bc51e14488b4f1810c7b69e9bd" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 35847 } } peers { permanent_uuid: "ed471121813d4d1a9eb3f4e1c02da65b" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 40153 } } peers { permanent_uuid: "35ab6a8afa1a4639b35923c7d881a6c7" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 40583 } }
I20260926 07:33:25.073851 6207 catalog_manager.cc:5495] T 561a92b62f5e4870a8281e704ae2e4fa P 35ab6a8afa1a4639b35923c7d881a6c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 35ab6a8afa1a4639b35923c7d881a6c7 (127.22.53.193). New cstate: current_term: 1 leader_uuid: "35ab6a8afa1a4639b35923c7d881a6c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a0860bc51e14488b4f1810c7b69e9bd" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 35847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed471121813d4d1a9eb3f4e1c02da65b" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 40153 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35ab6a8afa1a4639b35923c7d881a6c7" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 40583 } health_report { overall_health: HEALTHY } } }
W20260926 07:33:25.104355 6668 tablet.cc:2369] T 561a92b62f5e4870a8281e704ae2e4fa P 5a0860bc51e14488b4f1810c7b69e9bd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:33:25.122243 6691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:25.118336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50916 (local address 127.22.53.193:40583)
0926 07:33:25.118622 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:25.118629 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:25.118644 (+ 15us) server_negotiation.cc:407] Connection header received
0926 07:33:25.118726 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:25.118728 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:25.118784 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:25.118907 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:33:25.119172 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:25.119745 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:25.120647 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:25.120941 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:25.121051 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:25.121361 (+ 310us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:33:25.121860 (+ 499us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:33:25.121962 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:25.122043 (+ 81us) server_negotiation.cc:299] Negotiation successful
0926 07:33:25.122099 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":127,"threads_started":1}
I20260926 07:33:25.126670 6489 raft_consensus.cc:1261] T 561a92b62f5e4870a8281e704ae2e4fa P ed471121813d4d1a9eb3f4e1c02da65b [term 1 FOLLOWER]: Refusing update from remote peer 35ab6a8afa1a4639b35923c7d881a6c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:33:25.126672 6621 raft_consensus.cc:1261] T 561a92b62f5e4870a8281e704ae2e4fa P 5a0860bc51e14488b4f1810c7b69e9bd [term 1 FOLLOWER]: Refusing update from remote peer 35ab6a8afa1a4639b35923c7d881a6c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:33:25.127394 6403 consensus_queue.cc:1035] T 561a92b62f5e4870a8281e704ae2e4fa P 35ab6a8afa1a4639b35923c7d881a6c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a0860bc51e14488b4f1810c7b69e9bd" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 35847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:33:25.127655 6688 consensus_queue.cc:1035] T 561a92b62f5e4870a8281e704ae2e4fa P 35ab6a8afa1a4639b35923c7d881a6c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed471121813d4d1a9eb3f4e1c02da65b" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 40153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:33:25.145501 6694 mvcc.cc:205] Tried to move back new op lower bound from 7333511188991533056 to 7333511188770873344. Current Snapshot: MvccSnapshot[applied={T|T < 7333511188991533056 or (T in {7333511188991533056})}]
I20260926 07:33:25.146762 6209 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45718:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:33:25.146932 6209 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45718:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:33:25.149789 6209 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 788e7a47032d4f458d5493fcf0021f9d: Sending DeleteTablet for 3 replicas of tablet 561a92b62f5e4870a8281e704ae2e4fa
I20260926 07:33:25.150770 6331 tablet_service.cc:1514] Processing DeleteTablet for tablet 561a92b62f5e4870a8281e704ae2e4fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:33:25 UTC) from {username='kudu', principal='kudu/127.22.53.254@KRBTEST.COM'} at 127.0.0.1:50912
I20260926 07:33:25.150852 6601 tablet_service.cc:1514] Processing DeleteTablet for tablet 561a92b62f5e4870a8281e704ae2e4fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:33:25 UTC) from {username='kudu', principal='kudu/127.22.53.254@KRBTEST.COM'} at 127.0.0.1:54646
I20260926 07:33:25.150990 6469 tablet_service.cc:1514] Processing DeleteTablet for tablet 561a92b62f5e4870a8281e704ae2e4fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:33:25 UTC) from {username='kudu', principal='kudu/127.22.53.254@KRBTEST.COM'} at 127.0.0.1:59148
I20260926 07:33:25.151779 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 6247
I20260926 07:33:25.151811 6702 tablet_replica.cc:330] T 561a92b62f5e4870a8281e704ae2e4fa P 35ab6a8afa1a4639b35923c7d881a6c7: stopping tablet replica
I20260926 07:33:25.151844 6703 tablet_replica.cc:330] T 561a92b62f5e4870a8281e704ae2e4fa P ed471121813d4d1a9eb3f4e1c02da65b: stopping tablet replica
I20260926 07:33:25.151969 6702 raft_consensus.cc:2229] T 561a92b62f5e4870a8281e704ae2e4fa P 35ab6a8afa1a4639b35923c7d881a6c7 [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:33:25.152055 6704 tablet_replica.cc:330] T 561a92b62f5e4870a8281e704ae2e4fa P 5a0860bc51e14488b4f1810c7b69e9bd: stopping tablet replica
I20260926 07:33:25.152856 6704 raft_consensus.cc:2229] T 561a92b62f5e4870a8281e704ae2e4fa P 5a0860bc51e14488b4f1810c7b69e9bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:33:25.152848 6703 raft_consensus.cc:2229] T 561a92b62f5e4870a8281e704ae2e4fa P ed471121813d4d1a9eb3f4e1c02da65b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:33:25.153250 6703 raft_consensus.cc:2258] T 561a92b62f5e4870a8281e704ae2e4fa P ed471121813d4d1a9eb3f4e1c02da65b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:33:25.153249 6704 raft_consensus.cc:2258] T 561a92b62f5e4870a8281e704ae2e4fa P 5a0860bc51e14488b4f1810c7b69e9bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:33:25.155081 6703 ts_tablet_manager.cc:1905] T 561a92b62f5e4870a8281e704ae2e4fa P ed471121813d4d1a9eb3f4e1c02da65b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:33:25.155346 6704 ts_tablet_manager.cc:1905] T 561a92b62f5e4870a8281e704ae2e4fa P 5a0860bc51e14488b4f1810c7b69e9bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:33:25.157881 6704 ts_tablet_manager.cc:1918] T 561a92b62f5e4870a8281e704ae2e4fa P 5a0860bc51e14488b4f1810c7b69e9bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:33:25.158062 6704 log.cc:1196] T 561a92b62f5e4870a8281e704ae2e4fa P 5a0860bc51e14488b4f1810c7b69e9bd: Deleting WAL directory at /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-2/wal/wals/561a92b62f5e4870a8281e704ae2e4fa
I20260926 07:33:25.158535 6704 ts_tablet_manager.cc:1939] T 561a92b62f5e4870a8281e704ae2e4fa P 5a0860bc51e14488b4f1810c7b69e9bd: Deleting consensus metadata
I20260926 07:33:25.159089 6703 ts_tablet_manager.cc:1918] T 561a92b62f5e4870a8281e704ae2e4fa P ed471121813d4d1a9eb3f4e1c02da65b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:33:25.159214 6703 log.cc:1196] T 561a92b62f5e4870a8281e704ae2e4fa P ed471121813d4d1a9eb3f4e1c02da65b: Deleting WAL directory at /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-1/wal/wals/561a92b62f5e4870a8281e704ae2e4fa
I20260926 07:33:25.159659 6703 ts_tablet_manager.cc:1939] T 561a92b62f5e4870a8281e704ae2e4fa P ed471121813d4d1a9eb3f4e1c02da65b: Deleting consensus metadata
I20260926 07:33:25.160120 6195 catalog_manager.cc:4841] TS 5a0860bc51e14488b4f1810c7b69e9bd (127.22.53.195:35847): tablet 561a92b62f5e4870a8281e704ae2e4fa (table test-table [id=791126f3a5ad467a83e3c553b200a4b8]) successfully deleted
I20260926 07:33:25.161078 6193 catalog_manager.cc:4841] TS ed471121813d4d1a9eb3f4e1c02da65b (127.22.53.194:40153): tablet 561a92b62f5e4870a8281e704ae2e4fa (table test-table [id=791126f3a5ad467a83e3c553b200a4b8]) successfully deleted
W20260926 07:33:25.161119 6194 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.53.193:40583 (error 108)
I20260926 07:33:25.162030 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 6401
I20260926 07:33:25.162402 6677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:25.162030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.193:40583 (local address 127.0.0.1:50920)
0926 07:33:25.162193 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:25.162281 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.193:40583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260926 07:33:25.162865 6194 catalog_manager.cc:4584] TS 35ab6a8afa1a4639b35923c7d881a6c7 (127.22.53.193:40583): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 561a92b62f5e4870a8281e704ae2e4fa: Network error: Client connection negotiation failed: client connection to 127.22.53.193:40583: connect: Connection refused (error 111)
I20260926 07:33:25.168743 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 6538
I20260926 07:33:25.174948 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 6176
I20260926 07:33:45.182885 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.53.254:32891
--webserver_interface=127.22.53.254
--webserver_port=37697
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab
--principal=kudu/127.22.53.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:36403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.53.254:32891
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:33:45.302757 6706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:33:45.303158 6706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:33:45.303243 6706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:33:45.307788 6706 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 07:33:45.307914 6706 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 07:33:45.307943 6706 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 07:33:45.307967 6706 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:33:45.308004 6706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:33:45.308048 6706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:33:45.308099 6706 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:33:45.308139 6706 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:33:45.314137 6706 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:36403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.22.53.254:32891
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab
--principal=kudu/127.22.53.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.53.254:32891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.53.254
--webserver_port=37697
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6706
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:33:45.315409 6706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:33:45.316555 6706 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:33:45.326879 6714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:45.327045 6712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:33:45.327286 6706 server_base.cc:1034] running on GCE node
W20260926 07:33:45.326879 6711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:33:45.328248 6706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:33:45.329635 6706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:33:45.330801 6706 hybrid_clock.cc:648] HybridClock initialized: now 1790408025330782 us; error 39 us; skew 500 ppm
Sep 26 07:33:45 dist-test-slave-wdjc krb5kdc[5629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408025, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:33:45.337561 6706 init.cc:377] Logged in from keytab as kudu/127.22.53.254@KRBTEST.COM (short username kudu)
I20260926 07:33:45.338693 6706 webserver.cc:460] Webserver started at http://127.22.53.254:37697/ using document root <none> and password file <none>
I20260926 07:33:45.339345 6706 fs_manager.cc:359] Metadata directory not provided
I20260926 07:33:45.339453 6706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:33:45.344029 6706 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 07:33:45.346978 6721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:45.348955 6706 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:33:45.349128 6706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/master-0/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/master-0/wal
uuid: "788e7a47032d4f458d5493fcf0021f9d"
format_stamp: "Formatted at 2026-09-26 07:33:02 on dist-test-slave-wdjc"
server_key: "299243d4b6966ad5336afd5e0b3334a4"
server_key_iv: "6e2a817fc2ea3438e7c194a94f3aabfa"
server_key_version: "encryptionkey@0"
I20260926 07:33:45.349835 6706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:33:45.361196 6706 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:33:45.364692 6706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:33:45.364877 6706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:33:45.372596 6706 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.254:32891
I20260926 07:33:45.372998 6773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.254:32891 every 8 connection(s)
I20260926 07:33:45.373850 6706 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
I20260926 07:33:45.378630 6774 sys_catalog.cc:263] Verifying existing consensus state
I20260926 07:33:45.380462 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 6706
I20260926 07:33:45.380632 6774 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 788e7a47032d4f458d5493fcf0021f9d: Bootstrap starting.
I20260926 07:33:45.381014 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.53.193:40583
--local_ip_for_outbound_sockets=127.22.53.193
--tserver_master_addrs=127.22.53.254:32891
--webserver_port=46431
--webserver_interface=127.22.53.193
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.193.keytab
--principal=kudu/127.22.53.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:36403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:33:45.399865 6774 log.cc:824] T 00000000000000000000000000000000 P 788e7a47032d4f458d5493fcf0021f9d: Log is configured to *not* fsync() on all Append() calls
I20260926 07:33:45.412366 6774 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 788e7a47032d4f458d5493fcf0021f9d: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 07:33:45.412876 6774 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 788e7a47032d4f458d5493fcf0021f9d: Bootstrap complete.
I20260926 07:33:45.417732 6774 raft_consensus.cc:348] T 00000000000000000000000000000000 P 788e7a47032d4f458d5493fcf0021f9d [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788e7a47032d4f458d5493fcf0021f9d" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 32891 } }
I20260926 07:33:45.418239 6774 raft_consensus.cc:729] T 00000000000000000000000000000000 P 788e7a47032d4f458d5493fcf0021f9d [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 788e7a47032d4f458d5493fcf0021f9d, State: Initialized, Role: FOLLOWER
I20260926 07:33:45.418835 6774 consensus_queue.cc:260] T 00000000000000000000000000000000 P 788e7a47032d4f458d5493fcf0021f9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788e7a47032d4f458d5493fcf0021f9d" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 32891 } }
I20260926 07:33:45.419008 6774 raft_consensus.cc:388] T 00000000000000000000000000000000 P 788e7a47032d4f458d5493fcf0021f9d [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:33:45.419117 6774 raft_consensus.cc:482] T 00000000000000000000000000000000 P 788e7a47032d4f458d5493fcf0021f9d [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:33:45.419251 6774 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 788e7a47032d4f458d5493fcf0021f9d [term 2 FOLLOWER]: Advancing to term 3
I20260926 07:33:45.421726 6774 raft_consensus.cc:504] T 00000000000000000000000000000000 P 788e7a47032d4f458d5493fcf0021f9d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788e7a47032d4f458d5493fcf0021f9d" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 32891 } }
I20260926 07:33:45.422138 6774 leader_election.cc:302] T 00000000000000000000000000000000 P 788e7a47032d4f458d5493fcf0021f9d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 788e7a47032d4f458d5493fcf0021f9d; no voters:
I20260926 07:33:45.422544 6774 leader_election.cc:288] T 00000000000000000000000000000000 P 788e7a47032d4f458d5493fcf0021f9d [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 07:33:45.422679 6778 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 788e7a47032d4f458d5493fcf0021f9d [term 3 FOLLOWER]: Leader election won for term 3
I20260926 07:33:45.422896 6778 raft_consensus.cc:686] T 00000000000000000000000000000000 P 788e7a47032d4f458d5493fcf0021f9d [term 3 LEADER]: Becoming Leader. State: Replica: 788e7a47032d4f458d5493fcf0021f9d, State: Running, Role: LEADER
I20260926 07:33:45.423283 6778 consensus_queue.cc:237] T 00000000000000000000000000000000 P 788e7a47032d4f458d5493fcf0021f9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788e7a47032d4f458d5493fcf0021f9d" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 32891 } }
I20260926 07:33:45.423894 6774 sys_catalog.cc:564] T 00000000000000000000000000000000 P 788e7a47032d4f458d5493fcf0021f9d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:33:45.425000 6780 sys_catalog.cc:455] T 00000000000000000000000000000000 P 788e7a47032d4f458d5493fcf0021f9d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 788e7a47032d4f458d5493fcf0021f9d. Latest consensus state: current_term: 3 leader_uuid: "788e7a47032d4f458d5493fcf0021f9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788e7a47032d4f458d5493fcf0021f9d" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 32891 } } }
I20260926 07:33:45.425166 6780 sys_catalog.cc:458] T 00000000000000000000000000000000 P 788e7a47032d4f458d5493fcf0021f9d [sys.catalog]: This master's current role is: LEADER
I20260926 07:33:45.425587 6787 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:33:45.425606 6779 sys_catalog.cc:455] T 00000000000000000000000000000000 P 788e7a47032d4f458d5493fcf0021f9d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "788e7a47032d4f458d5493fcf0021f9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788e7a47032d4f458d5493fcf0021f9d" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 32891 } } }
I20260926 07:33:45.425733 6779 sys_catalog.cc:458] T 00000000000000000000000000000000 P 788e7a47032d4f458d5493fcf0021f9d [sys.catalog]: This master's current role is: LEADER
I20260926 07:33:45.431124 6787 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:33:45.431892 6787 catalog_manager.cc:1259] Loaded cluster ID: 8716aa67f32a4bd89be30c51639b80e4
I20260926 07:33:45.431988 6787 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:33:45.433984 6787 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:33:45.434679 6787 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 788e7a47032d4f458d5493fcf0021f9d: Loaded TSK: 0
I20260926 07:33:45.434782 6787 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 788e7a47032d4f458d5493fcf0021f9d: Loaded TSK: 1
F20260926 07:33:45.435638 6787 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f7c079109ad google::LogMessage::Fail() at ??:0
@ 0x7f7c0791388a google::LogMessage::SendToLog() at ??:0
@ 0x7f7c079104de google::LogMessage::Flush() at ??:0
@ 0x7f7c07911a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f7c0a5d418d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f7c0a5ccf0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f7c122757bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f7c1224f99c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f7c12250178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f7c12291df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f7c122b082b std::function<>::operator()() at ??:0
@ 0x7f7c1224fb4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f7c12250f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f7c1224d579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f7c12291640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f7c09b623d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f7c09b62c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f7c09b643f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f7c09b53512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f7c0b1976db start_thread at ??:0
@ 0x7f7c0640e61f clone at ??:0
W20260926 07:33:45.547081 6776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:33:45.547400 6776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:33:45.547448 6776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:33:45.551877 6776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:33:45.551964 6776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:33:45.552029 6776 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.53.193
I20260926 07:33:45.558001 6776 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:36403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.193.keytab
--principal=kudu/127.22.53.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.53.193:40583
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.53.193
--webserver_port=46431
--tserver_master_addrs=127.22.53.254:32891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6776
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.53.193
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:33:45.559309 6776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:33:45.560436 6776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:33:45.571285 6822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:45.571378 6821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:45.571278 6824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:33:45.571529 6776 server_base.cc:1034] running on GCE node
I20260926 07:33:45.572173 6776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:33:45.573007 6776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:33:45.574239 6776 hybrid_clock.cc:648] HybridClock initialized: now 1790408025574211 us; error 57 us; skew 500 ppm
Sep 26 07:33:45 dist-test-slave-wdjc krb5kdc[5629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408025, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:33:45.577710 6776 init.cc:377] Logged in from keytab as kudu/127.22.53.193@KRBTEST.COM (short username kudu)
I20260926 07:33:45.578859 6776 webserver.cc:460] Webserver started at http://127.22.53.193:46431/ using document root <none> and password file <none>
I20260926 07:33:45.579531 6776 fs_manager.cc:359] Metadata directory not provided
I20260926 07:33:45.579644 6776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:33:45.584404 6776 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 07:33:45.587342 6831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:45.589094 6776 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:33:45.589295 6776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-0/wal
uuid: "35ab6a8afa1a4639b35923c7d881a6c7"
format_stamp: "Formatted at 2026-09-26 07:33:03 on dist-test-slave-wdjc"
server_key: "e1ffe45e46c039841ee805ddd4d70b4d"
server_key_iv: "d0b5410a929a21d0fd39ce354f1d6709"
server_key_version: "encryptionkey@0"
I20260926 07:33:45.589978 6776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:33:45.613652 6776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:33:45.617776 6776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:33:45.618070 6776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:33:45.618796 6776 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:33:45.620129 6838 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 07:33:45.626464 6776 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 07:33:45.626608 6776 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.000s sys 0.000s
I20260926 07:33:45.626703 6776 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 07:33:45.628851 6776 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 07:33:45.628944 6776 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260926 07:33:45.629287 6838 tablet_bootstrap.cc:492] T 561a92b62f5e4870a8281e704ae2e4fa P 35ab6a8afa1a4639b35923c7d881a6c7: Bootstrap starting.
I20260926 07:33:45.639966 6776 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.193:40583
I20260926 07:33:45.640237 6945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.193:40583 every 8 connection(s)
I20260926 07:33:45.641173 6776 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-0/data/info.pb
I20260926 07:33:45.650130 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 6776
I20260926 07:33:45.650632 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.53.194:40153
--local_ip_for_outbound_sockets=127.22.53.194
--tserver_master_addrs=127.22.53.254:32891
--webserver_port=36643
--webserver_interface=127.22.53.194
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.194.keytab
--principal=kudu/127.22.53.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:36403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:33:45.651368 6838 log.cc:824] T 561a92b62f5e4870a8281e704ae2e4fa P 35ab6a8afa1a4639b35923c7d881a6c7: Log is configured to *not* fsync() on all Append() calls
I20260926 07:33:45.662564 6838 tablet_bootstrap.cc:492] T 561a92b62f5e4870a8281e704ae2e4fa P 35ab6a8afa1a4639b35923c7d881a6c7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 07:33:45.663096 6838 tablet_bootstrap.cc:492] T 561a92b62f5e4870a8281e704ae2e4fa P 35ab6a8afa1a4639b35923c7d881a6c7: Bootstrap complete.
I20260926 07:33:45.664677 6838 ts_tablet_manager.cc:1397] T 561a92b62f5e4870a8281e704ae2e4fa P 35ab6a8afa1a4639b35923c7d881a6c7: Time spent bootstrapping tablet: real 0.036s user 0.022s sys 0.013s
I20260926 07:33:45.668246 6838 raft_consensus.cc:348] T 561a92b62f5e4870a8281e704ae2e4fa P 35ab6a8afa1a4639b35923c7d881a6c7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a0860bc51e14488b4f1810c7b69e9bd" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 35847 } } peers { permanent_uuid: "ed471121813d4d1a9eb3f4e1c02da65b" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 40153 } } peers { permanent_uuid: "35ab6a8afa1a4639b35923c7d881a6c7" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 40583 } }
I20260926 07:33:45.668905 6838 raft_consensus.cc:729] T 561a92b62f5e4870a8281e704ae2e4fa P 35ab6a8afa1a4639b35923c7d881a6c7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35ab6a8afa1a4639b35923c7d881a6c7, State: Initialized, Role: FOLLOWER
I20260926 07:33:45.669481 6838 consensus_queue.cc:260] T 561a92b62f5e4870a8281e704ae2e4fa P 35ab6a8afa1a4639b35923c7d881a6c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a0860bc51e14488b4f1810c7b69e9bd" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 35847 } } peers { permanent_uuid: "ed471121813d4d1a9eb3f4e1c02da65b" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 40153 } } peers { permanent_uuid: "35ab6a8afa1a4639b35923c7d881a6c7" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 40583 } }
I20260926 07:33:45.670749 6838 ts_tablet_manager.cc:1428] T 561a92b62f5e4870a8281e704ae2e4fa P 35ab6a8afa1a4639b35923c7d881a6c7: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260926 07:33:45.695331 6948 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:33:45.643746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:32891 (local address 127.22.53.193:48691)
0926 07:33:45.644461 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:45.644554 (+ 93us) client_negotiation.cc:174] Beginning negotiation
0926 07:33:45.645670 (+ 1116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:33:45.694378 (+ 48708us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:32891: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":156,"threads_started":1}
W20260926 07:33:45.695873 6946 heartbeater.cc:644] Failed to heartbeat to 127.22.53.254:32891 (0 consecutive failures): Network error: Failed to ping master at 127.22.53.254:32891: Client connection negotiation failed: client connection to 127.22.53.254:32891: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 07:33:45.696813 6948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:45.696443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:32891 (local address 127.22.53.193:33235)
0926 07:33:45.696676 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:45.696750 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:32891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260926 07:33:45.697562 6948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:45.697362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:32891 (local address 127.22.53.193:50431)
0926 07:33:45.697467 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:45.697513 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:32891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260926 07:33:45.697840 6946 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 07:33:45.778597 6950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:33:45.778903 6950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:33:45.779003 6950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:33:45.783578 6950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:33:45.783685 6950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:33:45.783813 6950 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.53.194
I20260926 07:33:45.789855 6950 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:36403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.194.keytab
--principal=kudu/127.22.53.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.53.194:40153
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.53.194
--webserver_port=36643
--tserver_master_addrs=127.22.53.254:32891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6950
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.53.194
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:33:45.791123 6950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:33:45.792717 6950 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:33:45.803380 6957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:45.803370 6956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:45.803426 6959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:33:45.803625 6950 server_base.cc:1034] running on GCE node
I20260926 07:33:45.804381 6950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:33:45.805210 6950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:33:45.806399 6950 hybrid_clock.cc:648] HybridClock initialized: now 1790408025806381 us; error 42 us; skew 500 ppm
Sep 26 07:33:45 dist-test-slave-wdjc krb5kdc[5629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408025, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:33:45.809826 6950 init.cc:377] Logged in from keytab as kudu/127.22.53.194@KRBTEST.COM (short username kudu)
I20260926 07:33:45.810905 6950 webserver.cc:460] Webserver started at http://127.22.53.194:36643/ using document root <none> and password file <none>
I20260926 07:33:45.811553 6950 fs_manager.cc:359] Metadata directory not provided
I20260926 07:33:45.811666 6950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:33:45.816152 6950 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 07:33:45.818943 6966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:45.820479 6950 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 07:33:45.820711 6950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-1/wal
uuid: "ed471121813d4d1a9eb3f4e1c02da65b"
format_stamp: "Formatted at 2026-09-26 07:33:03 on dist-test-slave-wdjc"
server_key: "106f575b8ff401addcc90eff49306219"
server_key_iv: "ba0883ae30d30c700a88b832a8a47804"
server_key_version: "encryptionkey@0"
I20260926 07:33:45.821353 6950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:33:45.842481 6950 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:33:45.846096 6950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:33:45.846357 6950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:33:45.847028 6950 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:33:45.847980 6950 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:33:45.848052 6950 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:45.848132 6950 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:33:45.848199 6950 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:45.857424 6950 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.194:40153
I20260926 07:33:45.857441 7079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.194:40153 every 8 connection(s)
I20260926 07:33:45.858829 6950 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-1/data/info.pb
I20260926 07:33:45.859580 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 6950
I20260926 07:33:45.860097 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.53.195:35847
--local_ip_for_outbound_sockets=127.22.53.195
--tserver_master_addrs=127.22.53.254:32891
--webserver_port=43537
--webserver_interface=127.22.53.195
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.195.keytab
--principal=kudu/127.22.53.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:36403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:33:45.863770 7083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:45.861546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:32891 (local address 127.22.53.194:55417)
0926 07:33:45.862148 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:45.862312 (+ 164us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:32891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":136,"threads_started":1}
W20260926 07:33:45.864444 7080 heartbeater.cc:644] Failed to heartbeat to 127.22.53.254:32891 (0 consecutive failures): Network error: Failed to ping master at 127.22.53.254:32891: Client connection negotiation failed: client connection to 127.22.53.254:32891: connect: Connection refused (error 111)
I20260926 07:33:45.865345 7083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:45.865059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:32891 (local address 127.22.53.194:44427)
0926 07:33:45.865230 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:45.865282 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:32891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260926 07:33:45.866117 7083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:45.865796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:32891 (local address 127.22.53.194:58447)
0926 07:33:45.865941 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:45.866012 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:32891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260926 07:33:45.866369 7080 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 07:33:45.982499 7082 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:33:45.982821 7082 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:33:45.982867 7082 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:33:45.987139 7082 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:33:45.987275 7082 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:33:45.987377 7082 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.53.195
I20260926 07:33:45.993249 7082 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:36403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.195.keytab
--principal=kudu/127.22.53.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.53.195:35847
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.53.195
--webserver_port=43537
--tserver_master_addrs=127.22.53.254:32891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7082
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.53.195
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:33:45.994527 7082 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:33:45.995617 7082 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:33:46.006109 7091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:33:46.006153 7088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:33:46.006364 7082 server_base.cc:1034] running on GCE node
W20260926 07:33:46.006146 7089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:33:46.006999 7082 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:33:46.007822 7082 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:33:46.009037 7082 hybrid_clock.cc:648] HybridClock initialized: now 1790408026009015 us; error 39 us; skew 500 ppm
Sep 26 07:33:46 dist-test-slave-wdjc krb5kdc[5629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408026, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:33:46.012092 7082 init.cc:377] Logged in from keytab as kudu/127.22.53.195@KRBTEST.COM (short username kudu)
I20260926 07:33:46.013214 7082 webserver.cc:460] Webserver started at http://127.22.53.195:43537/ using document root <none> and password file <none>
I20260926 07:33:46.013875 7082 fs_manager.cc:359] Metadata directory not provided
I20260926 07:33:46.013983 7082 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:33:46.018394 7082 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 07:33:46.021122 7098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:46.022820 7082 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 07:33:46.023005 7082 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-2/wal
uuid: "5a0860bc51e14488b4f1810c7b69e9bd"
format_stamp: "Formatted at 2026-09-26 07:33:03 on dist-test-slave-wdjc"
server_key: "38343cbcbaa2c117f57ac9160cdf2edf"
server_key_iv: "9b0dc03585259ea20641300e57bba641"
server_key_version: "encryptionkey@0"
I20260926 07:33:46.023660 7082 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:33:46.036996 7082 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:33:46.041008 7082 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:33:46.041280 7082 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:33:46.041967 7082 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:33:46.042889 7082 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:33:46.042972 7082 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:46.043042 7082 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:33:46.043094 7082 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:33:46.052798 7082 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.195:35847
I20260926 07:33:46.052826 7211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.195:35847 every 8 connection(s)
I20260926 07:33:46.053931 7082 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407783529717-22743-0/minicluster-data/ts-2/data/info.pb
I20260926 07:33:46.058130 7214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:46.056208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:32891 (local address 127.22.53.195:35849)
0926 07:33:46.056773 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:46.056905 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:32891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":114,"threads_started":1}
I20260926 07:33:46.058749 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 7082
W20260926 07:33:46.058833 7212 heartbeater.cc:644] Failed to heartbeat to 127.22.53.254:32891 (0 consecutive failures): Network error: Failed to ping master at 127.22.53.254:32891: Client connection negotiation failed: client connection to 127.22.53.254:32891: connect: Connection refused (error 111)
I20260926 07:33:46.059597 7214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:46.059296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:32891 (local address 127.22.53.195:59795)
0926 07:33:46.059438 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:46.059503 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:32891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260926 07:33:46.060264 7214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:46.060065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:32891 (local address 127.22.53.195:51855)
0926 07:33:46.060176 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:46.060218 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:32891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260926 07:33:46.060521 7212 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 07:33:46.699138 7216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:46.698642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:32891 (local address 127.22.53.193:32901)
0926 07:33:46.698940 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:46.699014 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:32891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":77,"threads_started":1}
I20260926 07:33:46.867573 7217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:46.867091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:32891 (local address 127.22.53.194:59763)
0926 07:33:46.867387 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:46.867455 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:32891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":78,"threads_started":1}
I20260926 07:33:47.061717 7218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:47.061203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:32891 (local address 127.22.53.195:33027)
0926 07:33:47.061516 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:47.061582 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:32891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":97,"threads_started":1}
I20260926 07:33:47.142215 7219 raft_consensus.cc:482] T 561a92b62f5e4870a8281e704ae2e4fa P 35ab6a8afa1a4639b35923c7d881a6c7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:33:47.142455 7219 raft_consensus.cc:504] T 561a92b62f5e4870a8281e704ae2e4fa P 35ab6a8afa1a4639b35923c7d881a6c7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a0860bc51e14488b4f1810c7b69e9bd" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 35847 } } peers { permanent_uuid: "ed471121813d4d1a9eb3f4e1c02da65b" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 40153 } } peers { permanent_uuid: "35ab6a8afa1a4639b35923c7d881a6c7" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 40583 } }
I20260926 07:33:47.143703 7219 leader_election.cc:288] T 561a92b62f5e4870a8281e704ae2e4fa P 35ab6a8afa1a4639b35923c7d881a6c7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5a0860bc51e14488b4f1810c7b69e9bd (127.22.53.195:35847), ed471121813d4d1a9eb3f4e1c02da65b (127.22.53.194:40153)
Sep 26 07:33:47 dist-test-slave-wdjc krb5kdc[5629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408025, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.193@KRBTEST.COM for kudu/127.22.53.194@KRBTEST.COM
Sep 26 07:33:47 dist-test-slave-wdjc krb5kdc[5629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408025, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.193@KRBTEST.COM for kudu/127.22.53.195@KRBTEST.COM
I20260926 07:33:47.169102 7216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:47.144010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.195:35847 (local address 127.22.53.193:59455)
0926 07:33:47.144244 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:47.144259 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:33:47.144374 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:33:47.146674 (+ 2300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:33:47.146682 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:33:47.147786 (+ 1104us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:33:47.148702 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:47.148727 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:47.150840 (+ 2113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:47.150845 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:47.152677 (+ 1832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:47.152693 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:47.153812 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:47.153817 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:47.154123 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:47.159628 (+ 5505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:33:47.159658 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:33:47.162239 (+ 2581us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:33:47.165396 (+ 3157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:33:47.165404 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:33:47.165422 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:33:47.166005 (+ 583us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:33:47.166553 (+ 548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:33:47.166557 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:33:47.166559 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:33:47.166836 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:33:47.167719 (+ 883us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:33:47.167727 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:33:47.168112 (+ 385us) client_negotiation.cc:769] Sending connection context
0926 07:33:47.168386 (+ 274us) client_negotiation.cc:240] Negotiation successful
0926 07:33:47.168892 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"mutex_wait_us":577,"spinlock_wait_cycles":51968}
I20260926 07:33:47.169127 7221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:47.144189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.193:37203 (local address 127.22.53.194:40153)
0926 07:33:47.144634 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:47.144661 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:47.144709 (+ 48us) server_negotiation.cc:407] Connection header received
0926 07:33:47.145658 (+ 949us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:47.145668 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:47.146151 (+ 483us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:47.146521 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:47.148976 (+ 2455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:47.150597 (+ 1621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:47.152843 (+ 2246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:47.153629 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:47.153822 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:47.162429 (+ 8607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:47.162440 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:47.162459 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:47.162510 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:47.165196 (+ 2686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:47.166179 (+ 983us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:47.166186 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:47.166192 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:47.166403 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:47.166960 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:47.166963 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:47.166965 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:47.167572 (+ 607us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:47.167720 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:47.168602 (+ 882us) server_negotiation.cc:299] Negotiation successful
0926 07:33:47.168936 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":86,"threads_started":1}
I20260926 07:33:47.169629 7220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:47.144017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.194:40153 (local address 127.22.53.193:37203)
0926 07:33:47.144449 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:47.144464 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:33:47.144600 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:33:47.146674 (+ 2074us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:33:47.146682 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:33:47.147786 (+ 1104us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:33:47.148702 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:47.148727 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:47.150840 (+ 2113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:47.150845 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:47.152677 (+ 1832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:33:47.152693 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:33:47.153812 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:33:47.153817 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:33:47.154126 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:47.159624 (+ 5498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:33:47.159675 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:33:47.162024 (+ 2349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:33:47.165396 (+ 3372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:33:47.165404 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:33:47.165422 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:33:47.166005 (+ 583us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:33:47.166553 (+ 548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:33:47.166557 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:33:47.166560 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:33:47.166836 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:33:47.167719 (+ 883us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:33:47.167727 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:33:47.168112 (+ 385us) client_negotiation.cc:769] Sending connection context
0926 07:33:47.168386 (+ 274us) client_negotiation.cc:240] Negotiation successful
0926 07:33:47.169462 (+ 1076us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"mutex_wait_us":1246,"spinlock_wait_cycles":77952,"thread_start_us":85,"threads_started":1}
I20260926 07:33:47.169714 7222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:47.144174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.193:59455 (local address 127.22.53.195:35847)
0926 07:33:47.144634 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0926 07:33:47.144667 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0926 07:33:47.144708 (+ 41us) server_negotiation.cc:407] Connection header received
0926 07:33:47.145668 (+ 960us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:33:47.145674 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:33:47.146151 (+ 477us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:33:47.146521 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:33:47.148977 (+ 2456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:47.150597 (+ 1620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:47.152857 (+ 2260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:33:47.153630 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:33:47.153830 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:33:47.162492 (+ 8662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:33:47.162497 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:33:47.162514 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:33:47.162551 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:33:47.165196 (+ 2645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:47.166179 (+ 983us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:47.166186 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:47.166192 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:47.166403 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:33:47.166960 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:33:47.166963 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:33:47.166965 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:33:47.167572 (+ 607us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:33:47.167720 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 07:33:47.168914 (+ 1194us) server_negotiation.cc:299] Negotiation successful
0926 07:33:47.169546 (+ 632us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":96,"threads_started":1}
W20260926 07:33:47.172468 6832 leader_election.cc:341] T 561a92b62f5e4870a8281e704ae2e4fa P 35ab6a8afa1a4639b35923c7d881a6c7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ed471121813d4d1a9eb3f4e1c02da65b (127.22.53.194:40153): Not found: Tablet not found: 561a92b62f5e4870a8281e704ae2e4fa
W20260926 07:33:47.172701 6834 leader_election.cc:341] T 561a92b62f5e4870a8281e704ae2e4fa P 35ab6a8afa1a4639b35923c7d881a6c7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5a0860bc51e14488b4f1810c7b69e9bd (127.22.53.195:35847): Not found: Tablet not found: 561a92b62f5e4870a8281e704ae2e4fa
I20260926 07:33:47.172851 6834 leader_election.cc:302] T 561a92b62f5e4870a8281e704ae2e4fa P 35ab6a8afa1a4639b35923c7d881a6c7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 35ab6a8afa1a4639b35923c7d881a6c7; no voters: 5a0860bc51e14488b4f1810c7b69e9bd, ed471121813d4d1a9eb3f4e1c02da65b
I20260926 07:33:47.173156 7219 raft_consensus.cc:2726] T 561a92b62f5e4870a8281e704ae2e4fa P 35ab6a8afa1a4639b35923c7d881a6c7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 07:33:47.700486 7223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:47.700032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:32891 (local address 127.22.53.193:45959)
0926 07:33:47.700322 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:47.700386 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:32891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":81,"threads_started":1}
I20260926 07:33:47.869168 7224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:47.868609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:32891 (local address 127.22.53.194:56157)
0926 07:33:47.868988 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:47.869056 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:32891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":85,"threads_started":1}
I20260926 07:33:48.062894 7225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:48.062509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:32891 (local address 127.22.53.195:55159)
0926 07:33:48.062747 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:48.062799 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:32891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":70,"threads_started":1}
I20260926 07:33:48.701931 7226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:48.701451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:32891 (local address 127.22.53.193:57961)
0926 07:33:48.701740 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:48.701815 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:32891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":80,"threads_started":1}
I20260926 07:33:48.870576 7227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:48.870112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:32891 (local address 127.22.53.194:39757)
0926 07:33:48.870401 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:48.870465 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:32891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":81,"threads_started":1}
I20260926 07:33:49.064394 7228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:49.063877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:32891 (local address 127.22.53.195:46053)
0926 07:33:49.064223 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:49.064293 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:32891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":92,"threads_started":1}
I20260926 07:33:49.192659 7229 raft_consensus.cc:482] T 561a92b62f5e4870a8281e704ae2e4fa P 35ab6a8afa1a4639b35923c7d881a6c7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:33:49.192834 7229 raft_consensus.cc:504] T 561a92b62f5e4870a8281e704ae2e4fa P 35ab6a8afa1a4639b35923c7d881a6c7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a0860bc51e14488b4f1810c7b69e9bd" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 35847 } } peers { permanent_uuid: "ed471121813d4d1a9eb3f4e1c02da65b" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 40153 } } peers { permanent_uuid: "35ab6a8afa1a4639b35923c7d881a6c7" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 40583 } }
I20260926 07:33:49.193323 7229 leader_election.cc:288] T 561a92b62f5e4870a8281e704ae2e4fa P 35ab6a8afa1a4639b35923c7d881a6c7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5a0860bc51e14488b4f1810c7b69e9bd (127.22.53.195:35847), ed471121813d4d1a9eb3f4e1c02da65b (127.22.53.194:40153)
W20260926 07:33:49.194205 6834 leader_election.cc:341] T 561a92b62f5e4870a8281e704ae2e4fa P 35ab6a8afa1a4639b35923c7d881a6c7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5a0860bc51e14488b4f1810c7b69e9bd (127.22.53.195:35847): Not found: Tablet not found: 561a92b62f5e4870a8281e704ae2e4fa
W20260926 07:33:49.194334 6832 leader_election.cc:341] T 561a92b62f5e4870a8281e704ae2e4fa P 35ab6a8afa1a4639b35923c7d881a6c7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ed471121813d4d1a9eb3f4e1c02da65b (127.22.53.194:40153): Not found: Tablet not found: 561a92b62f5e4870a8281e704ae2e4fa
I20260926 07:33:49.194432 6832 leader_election.cc:302] T 561a92b62f5e4870a8281e704ae2e4fa P 35ab6a8afa1a4639b35923c7d881a6c7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 35ab6a8afa1a4639b35923c7d881a6c7; no voters: 5a0860bc51e14488b4f1810c7b69e9bd, ed471121813d4d1a9eb3f4e1c02da65b
I20260926 07:33:49.194588 7229 raft_consensus.cc:2726] T 561a92b62f5e4870a8281e704ae2e4fa P 35ab6a8afa1a4639b35923c7d881a6c7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 07:33:49.703289 7230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:49.702823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:32891 (local address 127.22.53.193:34563)
0926 07:33:49.703121 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:49.703186 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:32891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":82,"threads_started":1}
I20260926 07:33:49.872051 7231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:49.871581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:32891 (local address 127.22.53.194:46319)
0926 07:33:49.871884 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:49.871948 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:32891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":85,"threads_started":1}
I20260926 07:33:50.065877 7232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:50.065426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:32891 (local address 127.22.53.195:40245)
0926 07:33:50.065716 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:50.065780 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:32891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":79,"threads_started":1}
I20260926 07:33:50.704965 7233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:50.704385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:32891 (local address 127.22.53.193:53071)
0926 07:33:50.704802 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:50.704867 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:32891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":76,"threads_started":1}
I20260926 07:33:50.873665 7234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:50.873194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:32891 (local address 127.22.53.194:36265)
0926 07:33:50.873489 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:50.873561 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:32891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":102,"threads_started":1}
I20260926 07:33:51.067272 7235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:51.066832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:32891 (local address 127.22.53.195:48073)
0926 07:33:51.067109 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:51.067171 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:32891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":78,"threads_started":1}
I20260926 07:33:51.315379 7236 raft_consensus.cc:482] T 561a92b62f5e4870a8281e704ae2e4fa P 35ab6a8afa1a4639b35923c7d881a6c7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:33:51.315522 7236 raft_consensus.cc:504] T 561a92b62f5e4870a8281e704ae2e4fa P 35ab6a8afa1a4639b35923c7d881a6c7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a0860bc51e14488b4f1810c7b69e9bd" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 35847 } } peers { permanent_uuid: "ed471121813d4d1a9eb3f4e1c02da65b" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 40153 } } peers { permanent_uuid: "35ab6a8afa1a4639b35923c7d881a6c7" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 40583 } }
I20260926 07:33:51.316063 7236 leader_election.cc:288] T 561a92b62f5e4870a8281e704ae2e4fa P 35ab6a8afa1a4639b35923c7d881a6c7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5a0860bc51e14488b4f1810c7b69e9bd (127.22.53.195:35847), ed471121813d4d1a9eb3f4e1c02da65b (127.22.53.194:40153)
W20260926 07:33:51.317008 6832 leader_election.cc:341] T 561a92b62f5e4870a8281e704ae2e4fa P 35ab6a8afa1a4639b35923c7d881a6c7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ed471121813d4d1a9eb3f4e1c02da65b (127.22.53.194:40153): Not found: Tablet not found: 561a92b62f5e4870a8281e704ae2e4fa
W20260926 07:33:51.317126 6834 leader_election.cc:341] T 561a92b62f5e4870a8281e704ae2e4fa P 35ab6a8afa1a4639b35923c7d881a6c7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5a0860bc51e14488b4f1810c7b69e9bd (127.22.53.195:35847): Not found: Tablet not found: 561a92b62f5e4870a8281e704ae2e4fa
I20260926 07:33:51.317277 6834 leader_election.cc:302] T 561a92b62f5e4870a8281e704ae2e4fa P 35ab6a8afa1a4639b35923c7d881a6c7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 35ab6a8afa1a4639b35923c7d881a6c7; no voters: 5a0860bc51e14488b4f1810c7b69e9bd, ed471121813d4d1a9eb3f4e1c02da65b
I20260926 07:33:51.317482 7236 raft_consensus.cc:2726] T 561a92b62f5e4870a8281e704ae2e4fa P 35ab6a8afa1a4639b35923c7d881a6c7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 07:33:51.706386 7237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:51.705892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:32891 (local address 127.22.53.193:50501)
0926 07:33:51.706197 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:51.706283 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:32891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":80,"threads_started":1}
I20260926 07:33:51.875065 7238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:51.874595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:32891 (local address 127.22.53.194:53391)
0926 07:33:51.874896 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:51.874962 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:32891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":98,"threads_started":1}
I20260926 07:33:52.068818 7239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:52.068318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:32891 (local address 127.22.53.195:54591)
0926 07:33:52.068653 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:52.068721 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:32891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":94,"threads_started":1}
I20260926 07:33:52.707788 7240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:52.707322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:32891 (local address 127.22.53.193:46803)
0926 07:33:52.707604 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:52.707680 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:32891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":84,"threads_started":1}
I20260926 07:33:52.876729 7241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:52.876009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:32891 (local address 127.22.53.194:40031)
0926 07:33:52.876449 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:52.876525 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:32891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":131,"threads_started":1}
I20260926 07:33:53.070315 7242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:53.069783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:32891 (local address 127.22.53.195:59313)
0926 07:33:53.070148 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:53.070217 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:32891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":78,"threads_started":1}
I20260926 07:33:53.709124 7243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:53.708696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:32891 (local address 127.22.53.193:48159)
0926 07:33:53.708956 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:53.709014 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:32891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":75,"threads_started":1}
I20260926 07:33:53.878157 7244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:53.877664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:32891 (local address 127.22.53.194:59611)
0926 07:33:53.877980 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:53.878043 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:32891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":93,"threads_started":1}
I20260926 07:33:54.071689 7245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:54.071196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:32891 (local address 127.22.53.195:56973)
0926 07:33:54.071521 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:54.071588 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:32891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":91,"threads_started":1}
I20260926 07:33:54.710454 7246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:54.709996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:32891 (local address 127.22.53.193:60899)
0926 07:33:54.710291 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:54.710354 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:32891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":94,"threads_started":1}
I20260926 07:33:54.879444 7247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:54.878989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:32891 (local address 127.22.53.194:50005)
0926 07:33:54.879258 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:54.879337 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:32891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":93,"threads_started":1}
I20260926 07:33:54.985976 7248 raft_consensus.cc:482] T 561a92b62f5e4870a8281e704ae2e4fa P 35ab6a8afa1a4639b35923c7d881a6c7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:33:54.986119 7248 raft_consensus.cc:504] T 561a92b62f5e4870a8281e704ae2e4fa P 35ab6a8afa1a4639b35923c7d881a6c7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a0860bc51e14488b4f1810c7b69e9bd" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 35847 } } peers { permanent_uuid: "ed471121813d4d1a9eb3f4e1c02da65b" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 40153 } } peers { permanent_uuid: "35ab6a8afa1a4639b35923c7d881a6c7" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 40583 } }
I20260926 07:33:54.986716 7248 leader_election.cc:288] T 561a92b62f5e4870a8281e704ae2e4fa P 35ab6a8afa1a4639b35923c7d881a6c7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5a0860bc51e14488b4f1810c7b69e9bd (127.22.53.195:35847), ed471121813d4d1a9eb3f4e1c02da65b (127.22.53.194:40153)
W20260926 07:33:54.987524 6834 leader_election.cc:341] T 561a92b62f5e4870a8281e704ae2e4fa P 35ab6a8afa1a4639b35923c7d881a6c7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5a0860bc51e14488b4f1810c7b69e9bd (127.22.53.195:35847): Not found: Tablet not found: 561a92b62f5e4870a8281e704ae2e4fa
W20260926 07:33:54.987658 6832 leader_election.cc:341] T 561a92b62f5e4870a8281e704ae2e4fa P 35ab6a8afa1a4639b35923c7d881a6c7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ed471121813d4d1a9eb3f4e1c02da65b (127.22.53.194:40153): Not found: Tablet not found: 561a92b62f5e4870a8281e704ae2e4fa
I20260926 07:33:54.987746 6832 leader_election.cc:302] T 561a92b62f5e4870a8281e704ae2e4fa P 35ab6a8afa1a4639b35923c7d881a6c7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 35ab6a8afa1a4639b35923c7d881a6c7; no voters: 5a0860bc51e14488b4f1810c7b69e9bd, ed471121813d4d1a9eb3f4e1c02da65b
I20260926 07:33:54.987941 7248 raft_consensus.cc:2726] T 561a92b62f5e4870a8281e704ae2e4fa P 35ab6a8afa1a4639b35923c7d881a6c7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 07:33:55.073262 7249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:55.072764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:32891 (local address 127.22.53.195:56687)
0926 07:33:55.073081 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:55.073159 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:32891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":85,"threads_started":1}
I20260926 07:33:55.711989 7250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:55.711463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:32891 (local address 127.22.53.193:39859)
0926 07:33:55.711804 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:55.711881 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:32891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":97,"threads_started":1}
I20260926 07:33:55.881040 7251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:55.880477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:32891 (local address 127.22.53.194:58507)
0926 07:33:55.880864 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:55.880933 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:32891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":179,"threads_started":1}
I20260926 07:33:56.074676 7252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:56.074226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:32891 (local address 127.22.53.195:42631)
0926 07:33:56.074515 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:56.074577 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:32891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":105,"threads_started":1}
I20260926 07:33:56.713284 7253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:56.712858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:32891 (local address 127.22.53.193:48673)
0926 07:33:56.713126 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:56.713186 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:32891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":78,"threads_started":1}
I20260926 07:33:56.882423 7254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:56.881956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:32891 (local address 127.22.53.194:47071)
0926 07:33:56.882252 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:56.882321 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:32891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":78,"threads_started":1}
I20260926 07:33:57.076238 7255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:57.075717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:32891 (local address 127.22.53.195:38479)
0926 07:33:57.076052 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:57.076125 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:32891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":94,"threads_started":1}
I20260926 07:33:57.714622 7256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:57.714134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:32891 (local address 127.22.53.193:58719)
0926 07:33:57.714455 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:57.714515 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:32891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":118,"threads_started":1}
I20260926 07:33:57.883903 7257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:57.883330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:32891 (local address 127.22.53.194:56407)
0926 07:33:57.883693 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:57.883763 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:32891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":105,"threads_started":1}
I20260926 07:33:58.077816 7258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:58.077317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:32891 (local address 127.22.53.195:60605)
0926 07:33:58.077611 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:58.077697 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:32891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":84,"threads_started":1}
I20260926 07:33:58.716144 7259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:58.715638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:32891 (local address 127.22.53.193:41037)
0926 07:33:58.715975 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:58.716044 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:32891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":108,"threads_started":1}
I20260926 07:33:58.885598 7260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:58.885106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:32891 (local address 127.22.53.194:54843)
0926 07:33:58.885412 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:58.885482 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:32891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":80,"threads_started":1}
I20260926 07:33:59.079174 7261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:59.078716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:32891 (local address 127.22.53.195:44121)
0926 07:33:59.079009 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:59.079068 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:32891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":84,"threads_started":1}
I20260926 07:33:59.717633 7262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:59.717181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:32891 (local address 127.22.53.193:34071)
0926 07:33:59.717450 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:59.717518 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:32891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":85,"threads_started":1}
I20260926 07:33:59.887028 7263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:33:59.886534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:32891 (local address 127.22.53.194:53053)
0926 07:33:59.886853 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0926 07:33:59.886922 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:32891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":83,"threads_started":1}
I20260926 07:34:00.080704 7264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:00.080133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:32891 (local address 127.22.53.195:55471)
0926 07:34:00.080440 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:00.080509 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:32891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":83,"threads_started":1}
I20260926 07:34:00.719307 7265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:00.718826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:32891 (local address 127.22.53.193:41083)
0926 07:34:00.719105 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:00.719195 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:32891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":78,"threads_started":1}
I20260926 07:34:00.888612 7266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:00.888109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:32891 (local address 127.22.53.194:43167)
0926 07:34:00.888383 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:00.888453 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.254:32891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":121,"threads_started":1}
I20260926 07:34:01.065976 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 6776
I20260926 07:34:01.072278 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 6950
I20260926 07:34:01.078756 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 7082
2026-09-26T07:34:01Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59723 ms)
[----------] 30 tests from SecurityITest (257543 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407783529717-22743-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:34:01 dist-test-slave-wdjc krb5kdc[7269](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:34:01 dist-test-slave-wdjc krb5kdc[7269](info): set up 2 sockets
Sep 26 07:34:01 dist-test-slave-wdjc krb5kdc[7269](info): commencing operation
krb5kdc: starting...
W20260926 07:34:02.133093 22743 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407783529717-22743-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407783529717-22743-0/krb5kdc/test-user.keytab.
Sep 26 07:34:02 dist-test-slave-wdjc krb5kdc[7269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408042, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:34:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:34:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.53.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.53.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.53.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab.
Entry for principal kudu/127.22.53.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab.
I20260926 07:34:02.264053 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407783529717-22743-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407783529717-22743-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407783529717-22743-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.53.254:44877
--webserver_interface=127.22.53.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab
--principal=kudu/127.22.53.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:39989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.53.254:44877
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:34:02.381582 7282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:34:02.381978 7282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:34:02.382076 7282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:34:02.386365 7282 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:34:02.386490 7282 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:34:02.386528 7282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:34:02.386571 7282 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:34:02.386605 7282 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:34:02.392361 7282 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:39989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407783529717-22743-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407783529717-22743-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.53.254:44877
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab
--principal=kudu/127.22.53.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.53.254:44877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.53.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7282
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407783529717-22743-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:34:02.393608 7282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:34:02.394695 7282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:34:02.403506 7288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:02.403506 7287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:02.403507 7290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:34:02.403936 7282 server_base.cc:1034] running on GCE node
I20260926 07:34:02.404654 7282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:34:02.405894 7282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:34:02.407097 7282 hybrid_clock.cc:648] HybridClock initialized: now 1790408042407068 us; error 46 us; skew 500 ppm
Sep 26 07:34:02 dist-test-slave-wdjc krb5kdc[7269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408042, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:34:02.416304 7282 init.cc:377] Logged in from keytab as kudu/127.22.53.254@KRBTEST.COM (short username kudu)
I20260926 07:34:02.417546 7282 webserver.cc:460] Webserver started at http://127.22.53.254:35869/ using document root <none> and password file <none>
I20260926 07:34:02.418195 7282 fs_manager.cc:359] Metadata directory not provided
I20260926 07:34:02.418265 7282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:34:02.418499 7282 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:34:02.420882 7282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407783529717-22743-0/minicluster-data/master-0/data/instance:
uuid: "e168518ee12d4a5cb2ed5c24ed397d30"
format_stamp: "Formatted at 2026-09-26 07:34:02 on dist-test-slave-wdjc"
server_key: "485df3a6629f023433922b89062b90ce"
server_key_iv: "09477aa17ccafee847300a894943bf40"
server_key_version: "encryptionkey@0"
I20260926 07:34:02.421480 7282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407783529717-22743-0/minicluster-data/master-0/wal/instance:
uuid: "e168518ee12d4a5cb2ed5c24ed397d30"
format_stamp: "Formatted at 2026-09-26 07:34:02 on dist-test-slave-wdjc"
server_key: "485df3a6629f023433922b89062b90ce"
server_key_iv: "09477aa17ccafee847300a894943bf40"
server_key_version: "encryptionkey@0"
I20260926 07:34:02.425500 7282 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 07:34:02.428097 7297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:02.429524 7282 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:34:02.429728 7282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407783529717-22743-0/minicluster-data/master-0/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407783529717-22743-0/minicluster-data/master-0/wal
uuid: "e168518ee12d4a5cb2ed5c24ed397d30"
format_stamp: "Formatted at 2026-09-26 07:34:02 on dist-test-slave-wdjc"
server_key: "485df3a6629f023433922b89062b90ce"
server_key_iv: "09477aa17ccafee847300a894943bf40"
server_key_version: "encryptionkey@0"
I20260926 07:34:02.429872 7282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407783529717-22743-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407783529717-22743-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407783529717-22743-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:34:02.461061 7282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:34:02.464473 7282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:34:02.464728 7282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:34:02.472448 7282 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.254:44877
I20260926 07:34:02.472461 7349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.254:44877 every 8 connection(s)
I20260926 07:34:02.473559 7282 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
I20260926 07:34:02.476523 7350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:34:02.481755 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 7282
I20260926 07:34:02.481930 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407783529717-22743-0/minicluster-data/master-0/wal/instance
I20260926 07:34:02.482237 22743 external_mini_cluster.cc:1428] Setting key 6277d98c48b5281e19b801a32c01bae4
I20260926 07:34:02.482481 7350 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e168518ee12d4a5cb2ed5c24ed397d30: Bootstrap starting.
I20260926 07:34:02.485019 7350 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e168518ee12d4a5cb2ed5c24ed397d30: Neither blocks nor log segments found. Creating new log.
I20260926 07:34:02.485884 7350 log.cc:824] T 00000000000000000000000000000000 P e168518ee12d4a5cb2ed5c24ed397d30: Log is configured to *not* fsync() on all Append() calls
I20260926 07:34:02.487883 7350 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e168518ee12d4a5cb2ed5c24ed397d30: No bootstrap required, opened a new log
Sep 26 07:34:02 dist-test-slave-wdjc krb5kdc[7269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408042, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.53.254@KRBTEST.COM
I20260926 07:34:02.491717 7350 raft_consensus.cc:348] T 00000000000000000000000000000000 P e168518ee12d4a5cb2ed5c24ed397d30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e168518ee12d4a5cb2ed5c24ed397d30" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 44877 } }
I20260926 07:34:02.491946 7350 raft_consensus.cc:374] T 00000000000000000000000000000000 P e168518ee12d4a5cb2ed5c24ed397d30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:34:02.492034 7350 raft_consensus.cc:729] T 00000000000000000000000000000000 P e168518ee12d4a5cb2ed5c24ed397d30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e168518ee12d4a5cb2ed5c24ed397d30, State: Initialized, Role: FOLLOWER
I20260926 07:34:02.492498 7350 consensus_queue.cc:260] T 00000000000000000000000000000000 P e168518ee12d4a5cb2ed5c24ed397d30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e168518ee12d4a5cb2ed5c24ed397d30" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 44877 } }
I20260926 07:34:02.492702 7350 raft_consensus.cc:388] T 00000000000000000000000000000000 P e168518ee12d4a5cb2ed5c24ed397d30 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:34:02.492772 7350 raft_consensus.cc:482] T 00000000000000000000000000000000 P e168518ee12d4a5cb2ed5c24ed397d30 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:34:02.492878 7350 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e168518ee12d4a5cb2ed5c24ed397d30 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:34:02.493824 7350 raft_consensus.cc:504] T 00000000000000000000000000000000 P e168518ee12d4a5cb2ed5c24ed397d30 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e168518ee12d4a5cb2ed5c24ed397d30" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 44877 } }
I20260926 07:34:02.494199 7350 leader_election.cc:302] T 00000000000000000000000000000000 P e168518ee12d4a5cb2ed5c24ed397d30 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e168518ee12d4a5cb2ed5c24ed397d30; no voters:
I20260926 07:34:02.494565 7350 leader_election.cc:288] T 00000000000000000000000000000000 P e168518ee12d4a5cb2ed5c24ed397d30 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:34:02.494680 7355 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e168518ee12d4a5cb2ed5c24ed397d30 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:34:02.494933 7355 raft_consensus.cc:686] T 00000000000000000000000000000000 P e168518ee12d4a5cb2ed5c24ed397d30 [term 1 LEADER]: Becoming Leader. State: Replica: e168518ee12d4a5cb2ed5c24ed397d30, State: Running, Role: LEADER
I20260926 07:34:02.495275 7355 consensus_queue.cc:237] T 00000000000000000000000000000000 P e168518ee12d4a5cb2ed5c24ed397d30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e168518ee12d4a5cb2ed5c24ed397d30" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 44877 } }
I20260926 07:34:02.495988 7350 sys_catalog.cc:564] T 00000000000000000000000000000000 P e168518ee12d4a5cb2ed5c24ed397d30 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:34:02.497254 7357 sys_catalog.cc:455] T 00000000000000000000000000000000 P e168518ee12d4a5cb2ed5c24ed397d30 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e168518ee12d4a5cb2ed5c24ed397d30. Latest consensus state: current_term: 1 leader_uuid: "e168518ee12d4a5cb2ed5c24ed397d30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e168518ee12d4a5cb2ed5c24ed397d30" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 44877 } } }
I20260926 07:34:02.497255 7356 sys_catalog.cc:455] T 00000000000000000000000000000000 P e168518ee12d4a5cb2ed5c24ed397d30 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e168518ee12d4a5cb2ed5c24ed397d30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e168518ee12d4a5cb2ed5c24ed397d30" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 44877 } } }
I20260926 07:34:02.497416 7356 sys_catalog.cc:458] T 00000000000000000000000000000000 P e168518ee12d4a5cb2ed5c24ed397d30 [sys.catalog]: This master's current role is: LEADER
I20260926 07:34:02.497416 7357 sys_catalog.cc:458] T 00000000000000000000000000000000 P e168518ee12d4a5cb2ed5c24ed397d30 [sys.catalog]: This master's current role is: LEADER
I20260926 07:34:02.497620 7353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:02.483733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41728 (local address 127.22.53.254:44877)
0926 07:34:02.484158 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:02.484174 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:02.484208 (+ 34us) server_negotiation.cc:407] Connection header received
0926 07:34:02.484931 (+ 723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:02.484951 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:02.485329 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:02.485613 (+ 284us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:02.486542 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:02.487538 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:02.488469 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:02.488998 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:02.489174 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:02.491854 (+ 2680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:02.491859 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:02.491873 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:02.491901 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:02.493875 (+ 1974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:02.494342 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:02.494348 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:02.494354 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:02.494486 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:02.494868 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:02.494872 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:02.494875 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:02.495479 (+ 604us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:02.495666 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:02.495979 (+ 313us) server_negotiation.cc:299] Negotiation successful
0926 07:34:02.496463 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":137,"threads_started":1}
I20260926 07:34:02.498428 7364 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:34:02.502848 7364 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:34:02.508497 7364 catalog_manager.cc:1347] Generated new cluster ID: 4c20033878c6440086acb06ab1bc5f46
I20260926 07:34:02.508637 7364 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:34:02.530455 7364 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:34:02.531482 7364 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:34:02.538684 7364 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e168518ee12d4a5cb2ed5c24ed397d30: Generated new TSK 0
I20260926 07:34:02.539484 7364 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.53.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.53.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.53.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.193.keytab.
Entry for principal kudu/127.22.53.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.193.keytab.
I20260926 07:34:02.575928 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407783529717-22743-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407783529717-22743-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407783529717-22743-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407783529717-22743-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.53.193:0
--local_ip_for_outbound_sockets=127.22.53.193
--webserver_interface=127.22.53.193
--webserver_port=0
--tserver_master_addrs=127.22.53.254:44877
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.193.keytab
--principal=kudu/127.22.53.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:39989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:34:02.692739 7376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:34:02.693037 7376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:34:02.693084 7376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:34:02.697670 7376 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:34:02.697813 7376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:34:02.697916 7376 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.53.193
I20260926 07:34:02.703819 7376 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:39989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407783529717-22743-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407783529717-22743-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.193.keytab
--principal=kudu/127.22.53.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.53.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407783529717-22743-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.53.193
--webserver_port=0
--tserver_master_addrs=127.22.53.254:44877
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7376
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.53.193
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407783529717-22743-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:34:02.705044 7376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:34:02.706154 7376 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:34:02.716976 7381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:02.717108 7382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:34:02.717458 7376 server_base.cc:1034] running on GCE node
W20260926 07:34:02.716977 7384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:34:02.718007 7376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:34:02.718658 7376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:34:02.719874 7376 hybrid_clock.cc:648] HybridClock initialized: now 1790408042719857 us; error 31 us; skew 500 ppm
Sep 26 07:34:02 dist-test-slave-wdjc krb5kdc[7269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408042, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:34:02.723116 7376 init.cc:377] Logged in from keytab as kudu/127.22.53.193@KRBTEST.COM (short username kudu)
I20260926 07:34:02.724232 7376 webserver.cc:460] Webserver started at http://127.22.53.193:42885/ using document root <none> and password file <none>
I20260926 07:34:02.724939 7376 fs_manager.cc:359] Metadata directory not provided
I20260926 07:34:02.725016 7376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:34:02.725199 7376 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:34:02.727371 7376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407783529717-22743-0/minicluster-data/ts-0/data/instance:
uuid: "d034c7e75f4f4b5dadc738c2527359fe"
format_stamp: "Formatted at 2026-09-26 07:34:02 on dist-test-slave-wdjc"
server_key: "ec1cbe60ce0034e7c0a42ad6749d7fae"
server_key_iv: "b8179485fcb40d7b93eacc8ef1c7f8d8"
server_key_version: "encryptionkey@0"
I20260926 07:34:02.727922 7376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407783529717-22743-0/minicluster-data/ts-0/wal/instance:
uuid: "d034c7e75f4f4b5dadc738c2527359fe"
format_stamp: "Formatted at 2026-09-26 07:34:02 on dist-test-slave-wdjc"
server_key: "ec1cbe60ce0034e7c0a42ad6749d7fae"
server_key_iv: "b8179485fcb40d7b93eacc8ef1c7f8d8"
server_key_version: "encryptionkey@0"
I20260926 07:34:02.732185 7376 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 07:34:02.735289 7391 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:02.737044 7376 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 07:34:02.737178 7376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407783529717-22743-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407783529717-22743-0/minicluster-data/ts-0/wal
uuid: "d034c7e75f4f4b5dadc738c2527359fe"
format_stamp: "Formatted at 2026-09-26 07:34:02 on dist-test-slave-wdjc"
server_key: "ec1cbe60ce0034e7c0a42ad6749d7fae"
server_key_iv: "b8179485fcb40d7b93eacc8ef1c7f8d8"
server_key_version: "encryptionkey@0"
I20260926 07:34:02.737349 7376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407783529717-22743-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407783529717-22743-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407783529717-22743-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:34:02.756793 7376 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:34:02.760381 7376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:34:02.760643 7376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:34:02.761318 7376 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:34:02.762286 7376 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:34:02.762348 7376 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:02.762394 7376 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:34:02.762447 7376 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:02.772326 7376 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.193:34085
I20260926 07:34:02.772356 7504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.193:34085 every 8 connection(s)
I20260926 07:34:02.773401 7376 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407783529717-22743-0/minicluster-data/ts-0/data/info.pb
Sep 26 07:34:02 dist-test-slave-wdjc krb5kdc[7269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408042, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.193@KRBTEST.COM for kudu/127.22.53.254@KRBTEST.COM
I20260926 07:34:02.783516 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 7376
I20260926 07:34:02.783706 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407783529717-22743-0/minicluster-data/ts-0/wal/instance
I20260926 07:34:02.784018 22743 external_mini_cluster.cc:1428] Setting key c636944ae42a1ecdea8e00fc5eb75584
I20260926 07:34:02.788523 7353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:02.775370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.193:54149 (local address 127.22.53.254:44877)
0926 07:34:02.775570 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:02.775574 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:02.776443 (+ 869us) server_negotiation.cc:407] Connection header received
0926 07:34:02.777910 (+ 1467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:02.777914 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:02.777977 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:02.778059 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:02.779304 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:02.779869 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:02.781127 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:02.781392 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:02.781514 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:02.784296 (+ 2782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:02.784300 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:02.784303 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:02.784334 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:02.786077 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:02.786710 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:02.786714 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:02.786717 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:02.786821 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:02.787255 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:02.787261 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:02.787265 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:02.787476 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:02.787593 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:02.788189 (+ 596us) server_negotiation.cc:299] Negotiation successful
0926 07:34:02.788341 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260926 07:34:02.789427 7507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:02.775772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:44877 (local address 127.22.53.193:54149)
0926 07:34:02.776274 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:02.776321 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:02.777677 (+ 1356us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:02.778239 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:02.778248 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:02.778621 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:02.779135 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:02.779150 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:02.780018 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:02.780027 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:02.780985 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:02.780996 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:02.781564 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:02.781568 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:02.781747 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:02.782397 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:02.782413 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:02.784039 (+ 1626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:34:02.786241 (+ 2202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:02.786247 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:02.786263 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:02.786589 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:02.786942 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:02.786945 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:02.786947 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:02.787123 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:02.787598 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:34:02.787612 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:34:02.787903 (+ 291us) client_negotiation.cc:769] Sending connection context
0926 07:34:02.788119 (+ 216us) client_negotiation.cc:240] Negotiation successful
0926 07:34:02.788496 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":126,"threads_started":1}
I20260926 07:34:02.790820 7505 heartbeater.cc:347] Connected to a master server at 127.22.53.254:44877
I20260926 07:34:02.791100 7505 heartbeater.cc:464] Registering TS with master...
I20260926 07:34:02.791834 7505 heartbeater.cc:511] Master 127.22.53.254:44877 requested a full tablet report, sending...
I20260926 07:34:02.793632 7314 ts_manager.cc:194] Registered new tserver with Master: d034c7e75f4f4b5dadc738c2527359fe (127.22.53.193:34085)
WARNING: no policy specified for kudu/127.22.53.194@KRBTEST.COM; defaulting to no policy
I20260926 07:34:02.794854 7314 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.53.193@KRBTEST.COM'} at 127.22.53.193:54149
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.53.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.53.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.194.keytab.
Entry for principal kudu/127.22.53.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.194.keytab.
I20260926 07:34:02.813517 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407783529717-22743-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407783529717-22743-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407783529717-22743-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407783529717-22743-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.53.194:0
--local_ip_for_outbound_sockets=127.22.53.194
--webserver_interface=127.22.53.194
--webserver_port=0
--tserver_master_addrs=127.22.53.254:44877
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.194.keytab
--principal=kudu/127.22.53.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:39989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:34:02.929973 7510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:34:02.930261 7510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:34:02.930310 7510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:34:02.934559 7510 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:34:02.934640 7510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:34:02.934705 7510 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.53.194
I20260926 07:34:02.940414 7510 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:39989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407783529717-22743-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407783529717-22743-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.194.keytab
--principal=kudu/127.22.53.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.53.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407783529717-22743-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.53.194
--webserver_port=0
--tserver_master_addrs=127.22.53.254:44877
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7510
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.53.194
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407783529717-22743-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:34:02.941629 7510 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:34:02.942783 7510 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:34:02.953024 7515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:34:02.953038 7510 server_base.cc:1034] running on GCE node
W20260926 07:34:02.953058 7518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:02.953016 7516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:34:02.953730 7510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:34:02.954453 7510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:34:02.955701 7510 hybrid_clock.cc:648] HybridClock initialized: now 1790408042955651 us; error 68 us; skew 500 ppm
Sep 26 07:34:02 dist-test-slave-wdjc krb5kdc[7269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408042, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:34:02.958824 7510 init.cc:377] Logged in from keytab as kudu/127.22.53.194@KRBTEST.COM (short username kudu)
I20260926 07:34:02.959892 7510 webserver.cc:460] Webserver started at http://127.22.53.194:38249/ using document root <none> and password file <none>
I20260926 07:34:02.960574 7510 fs_manager.cc:359] Metadata directory not provided
I20260926 07:34:02.960683 7510 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:34:02.960930 7510 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:34:02.963348 7510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407783529717-22743-0/minicluster-data/ts-1/data/instance:
uuid: "7dc9fb4c264b4a148b42a8ab1cfc2d7b"
format_stamp: "Formatted at 2026-09-26 07:34:02 on dist-test-slave-wdjc"
server_key: "89d0760ffd7b68e924c849bf33699b79"
server_key_iv: "fb3ea66aaefa8c41ad7035431ee65a82"
server_key_version: "encryptionkey@0"
I20260926 07:34:02.964023 7510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407783529717-22743-0/minicluster-data/ts-1/wal/instance:
uuid: "7dc9fb4c264b4a148b42a8ab1cfc2d7b"
format_stamp: "Formatted at 2026-09-26 07:34:02 on dist-test-slave-wdjc"
server_key: "89d0760ffd7b68e924c849bf33699b79"
server_key_iv: "fb3ea66aaefa8c41ad7035431ee65a82"
server_key_version: "encryptionkey@0"
I20260926 07:34:02.968315 7510 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260926 07:34:02.971163 7525 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:02.972711 7510 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:34:02.972872 7510 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407783529717-22743-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407783529717-22743-0/minicluster-data/ts-1/wal
uuid: "7dc9fb4c264b4a148b42a8ab1cfc2d7b"
format_stamp: "Formatted at 2026-09-26 07:34:02 on dist-test-slave-wdjc"
server_key: "89d0760ffd7b68e924c849bf33699b79"
server_key_iv: "fb3ea66aaefa8c41ad7035431ee65a82"
server_key_version: "encryptionkey@0"
I20260926 07:34:02.973028 7510 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407783529717-22743-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407783529717-22743-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407783529717-22743-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:34:02.987933 7510 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:34:02.991532 7510 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:34:02.991756 7510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:34:02.992396 7510 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:34:02.993336 7510 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:34:02.993407 7510 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:02.993479 7510 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:34:02.993516 7510 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:03.003598 7510 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.194:46701
I20260926 07:34:03.003655 7638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.194:46701 every 8 connection(s)
I20260926 07:34:03.004665 7510 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407783529717-22743-0/minicluster-data/ts-1/data/info.pb
I20260926 07:34:03.010942 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 7510
I20260926 07:34:03.011101 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407783529717-22743-0/minicluster-data/ts-1/wal/instance
I20260926 07:34:03.011401 22743 external_mini_cluster.cc:1428] Setting key a3fa5c25d75142c30ee263951943b153
Sep 26 07:34:03 dist-test-slave-wdjc krb5kdc[7269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408042, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.194@KRBTEST.COM for kudu/127.22.53.254@KRBTEST.COM
I20260926 07:34:03.020469 7353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:03.006596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.194:33339 (local address 127.22.53.254:44877)
0926 07:34:03.006735 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:03.006739 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:03.007604 (+ 865us) server_negotiation.cc:407] Connection header received
0926 07:34:03.009113 (+ 1509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:03.009117 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:03.009184 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:03.009265 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:03.010551 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:03.011175 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:03.012570 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:03.012891 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:03.013012 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:03.015835 (+ 2823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:03.015848 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:03.015850 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:03.015883 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:03.017648 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:03.018470 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:03.018473 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:03.018475 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:03.018566 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:03.019070 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:03.019073 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:03.019074 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:03.019317 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:03.019411 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:03.020200 (+ 789us) server_negotiation.cc:299] Negotiation successful
0926 07:34:03.020330 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
WARNING: no policy specified for kudu/127.22.53.195@KRBTEST.COM; defaulting to no policy
I20260926 07:34:03.021922 7641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:03.006992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:44877 (local address 127.22.53.194:33339)
0926 07:34:03.007450 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:03.007494 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:03.008878 (+ 1384us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:03.009453 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:03.009462 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:03.009856 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:03.010386 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:03.010401 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:03.011316 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:03.011318 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:03.012404 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:03.012417 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:03.013062 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:03.013066 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:03.013245 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:03.013979 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:03.013996 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:03.015598 (+ 1602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:34:03.017817 (+ 2219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:03.017827 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:03.017853 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:03.018344 (+ 491us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:03.018681 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:03.018687 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:03.018691 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:03.018936 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:03.019438 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:34:03.019447 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:34:03.019869 (+ 422us) client_negotiation.cc:769] Sending connection context
0926 07:34:03.020119 (+ 250us) client_negotiation.cc:240] Negotiation successful
0926 07:34:03.020578 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":122,"threads_started":1}
I20260926 07:34:03.023360 7639 heartbeater.cc:347] Connected to a master server at 127.22.53.254:44877
I20260926 07:34:03.023645 7639 heartbeater.cc:464] Registering TS with master...
I20260926 07:34:03.024240 7639 heartbeater.cc:511] Master 127.22.53.254:44877 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.53.195@KRBTEST.COM" created.
I20260926 07:34:03.025751 7314 ts_manager.cc:194] Registered new tserver with Master: 7dc9fb4c264b4a148b42a8ab1cfc2d7b (127.22.53.194:46701)
I20260926 07:34:03.026669 7314 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.53.194@KRBTEST.COM'} at 127.22.53.194:33339
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.53.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.195.keytab.
Entry for principal kudu/127.22.53.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.195.keytab.
I20260926 07:34:03.044401 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407783529717-22743-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407783529717-22743-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407783529717-22743-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407783529717-22743-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.53.195:0
--local_ip_for_outbound_sockets=127.22.53.195
--webserver_interface=127.22.53.195
--webserver_port=0
--tserver_master_addrs=127.22.53.254:44877
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.195.keytab
--principal=kudu/127.22.53.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:39989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:34:03.158476 7644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:34:03.158797 7644 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:34:03.158844 7644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:34:03.163089 7644 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:34:03.163177 7644 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:34:03.163246 7644 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.53.195
I20260926 07:34:03.169418 7644 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:39989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407783529717-22743-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407783529717-22743-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.195.keytab
--principal=kudu/127.22.53.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.53.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407783529717-22743-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.53.195
--webserver_port=0
--tserver_master_addrs=127.22.53.254:44877
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7644
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.53.195
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407783529717-22743-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:34:03.170578 7644 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:34:03.171622 7644 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:34:03.182092 7650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:34:03.182242 7644 server_base.cc:1034] running on GCE node
W20260926 07:34:03.182101 7649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:03.182101 7652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:34:03.182862 7644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:34:03.183579 7644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:34:03.184746 7644 hybrid_clock.cc:648] HybridClock initialized: now 1790408043184725 us; error 34 us; skew 500 ppm
Sep 26 07:34:03 dist-test-slave-wdjc krb5kdc[7269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408043, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:34:03.187750 7644 init.cc:377] Logged in from keytab as kudu/127.22.53.195@KRBTEST.COM (short username kudu)
I20260926 07:34:03.188815 7644 webserver.cc:460] Webserver started at http://127.22.53.195:43405/ using document root <none> and password file <none>
I20260926 07:34:03.189441 7644 fs_manager.cc:359] Metadata directory not provided
I20260926 07:34:03.189507 7644 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:34:03.189736 7644 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:34:03.191895 7644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407783529717-22743-0/minicluster-data/ts-2/data/instance:
uuid: "d2a651b924ca4e28b2b7f0b38c482057"
format_stamp: "Formatted at 2026-09-26 07:34:03 on dist-test-slave-wdjc"
server_key: "5539dc958ef337e6420c1b03b2df7068"
server_key_iv: "b60234fc727eadcf695a79d776e40cdc"
server_key_version: "encryptionkey@0"
I20260926 07:34:03.192505 7644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407783529717-22743-0/minicluster-data/ts-2/wal/instance:
uuid: "d2a651b924ca4e28b2b7f0b38c482057"
format_stamp: "Formatted at 2026-09-26 07:34:03 on dist-test-slave-wdjc"
server_key: "5539dc958ef337e6420c1b03b2df7068"
server_key_iv: "b60234fc727eadcf695a79d776e40cdc"
server_key_version: "encryptionkey@0"
I20260926 07:34:03.196691 7644 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 07:34:03.199301 7659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:03.200855 7644 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:34:03.201004 7644 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407783529717-22743-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407783529717-22743-0/minicluster-data/ts-2/wal
uuid: "d2a651b924ca4e28b2b7f0b38c482057"
format_stamp: "Formatted at 2026-09-26 07:34:03 on dist-test-slave-wdjc"
server_key: "5539dc958ef337e6420c1b03b2df7068"
server_key_iv: "b60234fc727eadcf695a79d776e40cdc"
server_key_version: "encryptionkey@0"
I20260926 07:34:03.201159 7644 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407783529717-22743-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407783529717-22743-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407783529717-22743-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:34:03.215312 7644 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:34:03.218849 7644 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:34:03.219115 7644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:34:03.219774 7644 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:34:03.220783 7644 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:34:03.220858 7644 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:03.220934 7644 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:34:03.220983 7644 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:03.231416 7644 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.195:45009
I20260926 07:34:03.231464 7772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.195:45009 every 8 connection(s)
I20260926 07:34:03.232453 7644 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407783529717-22743-0/minicluster-data/ts-2/data/info.pb
I20260926 07:34:03.241722 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 7644
I20260926 07:34:03.241892 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407783529717-22743-0/minicluster-data/ts-2/wal/instance
I20260926 07:34:03.242204 22743 external_mini_cluster.cc:1428] Setting key 7f13f6bfa4d91dcc6826312998f55a42
Sep 26 07:34:03 dist-test-slave-wdjc krb5kdc[7269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408043, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.195@KRBTEST.COM for kudu/127.22.53.254@KRBTEST.COM
I20260926 07:34:03.247841 7353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:03.234550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.195:54553 (local address 127.22.53.254:44877)
0926 07:34:03.234752 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:03.234759 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:03.235563 (+ 804us) server_negotiation.cc:407] Connection header received
0926 07:34:03.237183 (+ 1620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:03.237187 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:03.237252 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:03.237357 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:03.238662 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:03.239232 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:03.240468 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:03.240790 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:03.240921 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:03.243995 (+ 3074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:03.243999 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:03.244002 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:03.244030 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:03.245685 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:03.246281 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:03.246284 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:03.246286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:03.246372 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:03.246719 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:03.246723 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:03.246725 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:03.246918 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:03.247023 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:03.247536 (+ 513us) server_negotiation.cc:299] Negotiation successful
0926 07:34:03.247677 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260926 07:34:03.248821 7775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:03.234934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:44877 (local address 127.22.53.195:54553)
0926 07:34:03.235401 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:03.235447 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:03.236941 (+ 1494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:03.237539 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:03.237547 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:03.237938 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:03.238482 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:03.238497 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:03.239385 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:03.239389 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:03.240320 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:03.240331 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:03.240967 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:03.240972 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:03.241165 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:03.242015 (+ 850us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:03.242033 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:03.243835 (+ 1802us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:34:03.245816 (+ 1981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:03.245823 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:03.245841 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:03.246165 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:03.246463 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:03.246467 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:03.246469 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:03.246623 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:03.247021 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:34:03.247029 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:34:03.247304 (+ 275us) client_negotiation.cc:769] Sending connection context
0926 07:34:03.247497 (+ 193us) client_negotiation.cc:240] Negotiation successful
0926 07:34:03.247854 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":125,"threads_started":1}
I20260926 07:34:03.250103 7773 heartbeater.cc:347] Connected to a master server at 127.22.53.254:44877
I20260926 07:34:03.250404 7773 heartbeater.cc:464] Registering TS with master...
I20260926 07:34:03.250944 7773 heartbeater.cc:511] Master 127.22.53.254:44877 requested a full tablet report, sending...
I20260926 07:34:03.252099 7314 ts_manager.cc:194] Registered new tserver with Master: d2a651b924ca4e28b2b7f0b38c482057 (127.22.53.195:45009)
I20260926 07:34:03.252760 7314 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.53.195@KRBTEST.COM'} at 127.22.53.195:54553
I20260926 07:34:03.256718 22743 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:34:03.266355 7353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:03.258838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47688 (local address 127.22.53.254:44877)
0926 07:34:03.258976 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:03.258981 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:03.259128 (+ 147us) server_negotiation.cc:407] Connection header received
0926 07:34:03.259239 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:03.259241 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:03.259294 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:03.259369 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:03.260100 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:03.260719 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:03.261536 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:03.261736 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:03.261813 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:03.263213 (+ 1400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:03.263216 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:03.263218 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:03.263243 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:03.264776 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:03.265174 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:03.265176 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:03.265177 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:03.265251 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:03.265520 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:03.265523 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:03.265524 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:03.265789 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:03.265911 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:03.266102 (+ 191us) server_negotiation.cc:299] Negotiation successful
0926 07:34:03.266205 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 07:34:03.269436 7314 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47688:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:34:03.272794 7314 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:34:03.288379 7787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:03.282509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41328 (local address 127.22.53.193:34085)
0926 07:34:03.282914 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:03.282930 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:03.282956 (+ 26us) server_negotiation.cc:407] Connection header received
0926 07:34:03.283159 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:03.283168 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:03.283427 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:03.283606 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:34:03.284221 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:03.285421 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:03.286854 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:03.287593 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:03.287704 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:03.287869 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:03.288091 (+ 222us) server_negotiation.cc:299] Negotiation successful
0926 07:34:03.288218 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":89,"threads_started":1}
I20260926 07:34:03.288378 7788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:03.282004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.193:34085 (local address 127.0.0.1:41328)
0926 07:34:03.282873 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:03.282888 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:03.282982 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:03.283641 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:03.283647 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:03.283685 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:34:03.284095 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:03.284101 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:03.285562 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:03.285565 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:03.286720 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:03.286730 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:03.287734 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:03.287738 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:03.287858 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:03.287910 (+ 52us) client_negotiation.cc:769] Sending connection context
0926 07:34:03.288040 (+ 130us) client_negotiation.cc:240] Negotiation successful
0926 07:34:03.288183 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":797,"mutex_wait_us":244,"spinlock_wait_cycles":4736,"thread_start_us":131,"threads_started":1}
I20260926 07:34:03.288856 7784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:03.282611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.195:45009 (local address 127.0.0.1:38796)
0926 07:34:03.283455 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:03.283469 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:03.283570 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:03.284049 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:03.284052 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:03.284071 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:34:03.284179 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:03.284189 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:03.285415 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:03.285418 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:03.287169 (+ 1751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:03.287181 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:03.288417 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:03.288421 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:03.288522 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:03.288581 (+ 59us) client_negotiation.cc:769] Sending connection context
0926 07:34:03.288656 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 07:34:03.288725 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":777,"mutex_wait_us":154,"thread_start_us":129,"threads_started":1}
I20260926 07:34:03.289059 7786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:03.281810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.194:46701 (local address 127.0.0.1:38960)
0926 07:34:03.282409 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:03.282467 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:03.282605 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:03.283645 (+ 1040us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:03.283648 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:03.283687 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:34:03.283881 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:03.283890 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:03.285421 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:03.285425 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:03.287169 (+ 1744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:03.287181 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:03.288297 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:03.288301 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:03.288398 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:03.288431 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 07:34:03.288487 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 07:34:03.288903 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414,"mutex_wait_us":126,"spinlock_wait_cycles":7680,"thread_start_us":99,"threads_started":1}
I20260926 07:34:03.289295 7789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:03.283077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38796 (local address 127.22.53.195:45009)
0926 07:34:03.283456 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:03.283468 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:03.283555 (+ 87us) server_negotiation.cc:407] Connection header received
0926 07:34:03.283742 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:03.283749 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:03.283910 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:03.284070 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:34:03.284315 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:03.285223 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:03.287320 (+ 2097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:03.288112 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:03.288214 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:03.288347 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:03.289032 (+ 685us) server_negotiation.cc:299] Negotiation successful
0926 07:34:03.289142 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":99,"threads_started":1}
I20260926 07:34:03.289742 7785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:03.281972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38960 (local address 127.22.53.194:46701)
0926 07:34:03.283162 (+ 1190us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:03.283176 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:03.283197 (+ 21us) server_negotiation.cc:407] Connection header received
0926 07:34:03.283279 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:03.283286 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:03.283489 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:03.283792 (+ 303us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:34:03.284004 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:03.285263 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:03.287336 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:03.288139 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:03.289257 (+ 1118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:03.289411 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:03.289468 (+ 57us) server_negotiation.cc:299] Negotiation successful
0926 07:34:03.289582 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1092,"thread_start_us":139,"threads_started":1}
I20260926 07:34:03.292054 7573 tablet_service.cc:1467] Processing CreateTablet for tablet dd8676c2dff9444293cf3aa0832728bd (DEFAULT_TABLE table=test-table [id=6e8d87d279c74fe2a34e8942845311aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:34:03.292069 7439 tablet_service.cc:1467] Processing CreateTablet for tablet dd8676c2dff9444293cf3aa0832728bd (DEFAULT_TABLE table=test-table [id=6e8d87d279c74fe2a34e8942845311aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:34:03.292074 7707 tablet_service.cc:1467] Processing CreateTablet for tablet dd8676c2dff9444293cf3aa0832728bd (DEFAULT_TABLE table=test-table [id=6e8d87d279c74fe2a34e8942845311aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:34:03.293184 7439 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd8676c2dff9444293cf3aa0832728bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:34:03.293184 7707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd8676c2dff9444293cf3aa0832728bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:34:03.293187 7573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd8676c2dff9444293cf3aa0832728bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:34:03.300343 7790 tablet_bootstrap.cc:492] T dd8676c2dff9444293cf3aa0832728bd P 7dc9fb4c264b4a148b42a8ab1cfc2d7b: Bootstrap starting.
I20260926 07:34:03.301640 7791 tablet_bootstrap.cc:492] T dd8676c2dff9444293cf3aa0832728bd P d034c7e75f4f4b5dadc738c2527359fe: Bootstrap starting.
I20260926 07:34:03.303205 7790 tablet_bootstrap.cc:654] T dd8676c2dff9444293cf3aa0832728bd P 7dc9fb4c264b4a148b42a8ab1cfc2d7b: Neither blocks nor log segments found. Creating new log.
I20260926 07:34:03.303373 7791 tablet_bootstrap.cc:654] T dd8676c2dff9444293cf3aa0832728bd P d034c7e75f4f4b5dadc738c2527359fe: Neither blocks nor log segments found. Creating new log.
I20260926 07:34:03.304304 7791 log.cc:824] T dd8676c2dff9444293cf3aa0832728bd P d034c7e75f4f4b5dadc738c2527359fe: Log is configured to *not* fsync() on all Append() calls
I20260926 07:34:03.304672 7790 log.cc:824] T dd8676c2dff9444293cf3aa0832728bd P 7dc9fb4c264b4a148b42a8ab1cfc2d7b: Log is configured to *not* fsync() on all Append() calls
I20260926 07:34:03.305020 7792 tablet_bootstrap.cc:492] T dd8676c2dff9444293cf3aa0832728bd P d2a651b924ca4e28b2b7f0b38c482057: Bootstrap starting.
I20260926 07:34:03.306631 7791 tablet_bootstrap.cc:492] T dd8676c2dff9444293cf3aa0832728bd P d034c7e75f4f4b5dadc738c2527359fe: No bootstrap required, opened a new log
I20260926 07:34:03.306631 7790 tablet_bootstrap.cc:492] T dd8676c2dff9444293cf3aa0832728bd P 7dc9fb4c264b4a148b42a8ab1cfc2d7b: No bootstrap required, opened a new log
I20260926 07:34:03.306843 7791 ts_tablet_manager.cc:1397] T dd8676c2dff9444293cf3aa0832728bd P d034c7e75f4f4b5dadc738c2527359fe: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260926 07:34:03.306843 7790 ts_tablet_manager.cc:1397] T dd8676c2dff9444293cf3aa0832728bd P 7dc9fb4c264b4a148b42a8ab1cfc2d7b: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260926 07:34:03.307399 7792 tablet_bootstrap.cc:654] T dd8676c2dff9444293cf3aa0832728bd P d2a651b924ca4e28b2b7f0b38c482057: Neither blocks nor log segments found. Creating new log.
I20260926 07:34:03.308647 7792 log.cc:824] T dd8676c2dff9444293cf3aa0832728bd P d2a651b924ca4e28b2b7f0b38c482057: Log is configured to *not* fsync() on all Append() calls
I20260926 07:34:03.310617 7792 tablet_bootstrap.cc:492] T dd8676c2dff9444293cf3aa0832728bd P d2a651b924ca4e28b2b7f0b38c482057: No bootstrap required, opened a new log
I20260926 07:34:03.310863 7792 ts_tablet_manager.cc:1397] T dd8676c2dff9444293cf3aa0832728bd P d2a651b924ca4e28b2b7f0b38c482057: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 07:34:03.311486 7791 raft_consensus.cc:348] T dd8676c2dff9444293cf3aa0832728bd P d034c7e75f4f4b5dadc738c2527359fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7dc9fb4c264b4a148b42a8ab1cfc2d7b" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 46701 } } peers { permanent_uuid: "d034c7e75f4f4b5dadc738c2527359fe" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 34085 } } peers { permanent_uuid: "d2a651b924ca4e28b2b7f0b38c482057" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 45009 } }
I20260926 07:34:03.311832 7791 raft_consensus.cc:374] T dd8676c2dff9444293cf3aa0832728bd P d034c7e75f4f4b5dadc738c2527359fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:34:03.311899 7791 raft_consensus.cc:729] T dd8676c2dff9444293cf3aa0832728bd P d034c7e75f4f4b5dadc738c2527359fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d034c7e75f4f4b5dadc738c2527359fe, State: Initialized, Role: FOLLOWER
I20260926 07:34:03.312398 7791 consensus_queue.cc:260] T dd8676c2dff9444293cf3aa0832728bd P d034c7e75f4f4b5dadc738c2527359fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7dc9fb4c264b4a148b42a8ab1cfc2d7b" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 46701 } } peers { permanent_uuid: "d034c7e75f4f4b5dadc738c2527359fe" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 34085 } } peers { permanent_uuid: "d2a651b924ca4e28b2b7f0b38c482057" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 45009 } }
I20260926 07:34:03.312304 7790 raft_consensus.cc:348] T dd8676c2dff9444293cf3aa0832728bd P 7dc9fb4c264b4a148b42a8ab1cfc2d7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7dc9fb4c264b4a148b42a8ab1cfc2d7b" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 46701 } } peers { permanent_uuid: "d034c7e75f4f4b5dadc738c2527359fe" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 34085 } } peers { permanent_uuid: "d2a651b924ca4e28b2b7f0b38c482057" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 45009 } }
I20260926 07:34:03.312582 7790 raft_consensus.cc:374] T dd8676c2dff9444293cf3aa0832728bd P 7dc9fb4c264b4a148b42a8ab1cfc2d7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:34:03.312640 7790 raft_consensus.cc:729] T dd8676c2dff9444293cf3aa0832728bd P 7dc9fb4c264b4a148b42a8ab1cfc2d7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7dc9fb4c264b4a148b42a8ab1cfc2d7b, State: Initialized, Role: FOLLOWER
I20260926 07:34:03.313199 7505 heartbeater.cc:503] Master 127.22.53.254:44877 was elected leader, sending a full tablet report...
I20260926 07:34:03.313213 7790 consensus_queue.cc:260] T dd8676c2dff9444293cf3aa0832728bd P 7dc9fb4c264b4a148b42a8ab1cfc2d7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7dc9fb4c264b4a148b42a8ab1cfc2d7b" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 46701 } } peers { permanent_uuid: "d034c7e75f4f4b5dadc738c2527359fe" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 34085 } } peers { permanent_uuid: "d2a651b924ca4e28b2b7f0b38c482057" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 45009 } }
I20260926 07:34:03.313988 7791 ts_tablet_manager.cc:1428] T dd8676c2dff9444293cf3aa0832728bd P d034c7e75f4f4b5dadc738c2527359fe: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260926 07:34:03.314424 7790 ts_tablet_manager.cc:1428] T dd8676c2dff9444293cf3aa0832728bd P 7dc9fb4c264b4a148b42a8ab1cfc2d7b: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260926 07:34:03.314633 7639 heartbeater.cc:503] Master 127.22.53.254:44877 was elected leader, sending a full tablet report...
I20260926 07:34:03.315092 7792 raft_consensus.cc:348] T dd8676c2dff9444293cf3aa0832728bd P d2a651b924ca4e28b2b7f0b38c482057 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7dc9fb4c264b4a148b42a8ab1cfc2d7b" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 46701 } } peers { permanent_uuid: "d034c7e75f4f4b5dadc738c2527359fe" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 34085 } } peers { permanent_uuid: "d2a651b924ca4e28b2b7f0b38c482057" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 45009 } }
I20260926 07:34:03.315382 7792 raft_consensus.cc:374] T dd8676c2dff9444293cf3aa0832728bd P d2a651b924ca4e28b2b7f0b38c482057 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:34:03.315444 7792 raft_consensus.cc:729] T dd8676c2dff9444293cf3aa0832728bd P d2a651b924ca4e28b2b7f0b38c482057 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2a651b924ca4e28b2b7f0b38c482057, State: Initialized, Role: FOLLOWER
I20260926 07:34:03.315922 7792 consensus_queue.cc:260] T dd8676c2dff9444293cf3aa0832728bd P d2a651b924ca4e28b2b7f0b38c482057 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7dc9fb4c264b4a148b42a8ab1cfc2d7b" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 46701 } } peers { permanent_uuid: "d034c7e75f4f4b5dadc738c2527359fe" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 34085 } } peers { permanent_uuid: "d2a651b924ca4e28b2b7f0b38c482057" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 45009 } }
I20260926 07:34:03.316689 7773 heartbeater.cc:503] Master 127.22.53.254:44877 was elected leader, sending a full tablet report...
I20260926 07:34:03.317148 7792 ts_tablet_manager.cc:1428] T dd8676c2dff9444293cf3aa0832728bd P d2a651b924ca4e28b2b7f0b38c482057: Time spent starting tablet: real 0.006s user 0.002s sys 0.006s
I20260926 07:34:03.401378 7798 raft_consensus.cc:482] T dd8676c2dff9444293cf3aa0832728bd P d2a651b924ca4e28b2b7f0b38c482057 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:34:03.401636 7798 raft_consensus.cc:504] T dd8676c2dff9444293cf3aa0832728bd P d2a651b924ca4e28b2b7f0b38c482057 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7dc9fb4c264b4a148b42a8ab1cfc2d7b" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 46701 } } peers { permanent_uuid: "d034c7e75f4f4b5dadc738c2527359fe" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 34085 } } peers { permanent_uuid: "d2a651b924ca4e28b2b7f0b38c482057" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 45009 } }
I20260926 07:34:03.402865 7798 leader_election.cc:288] T dd8676c2dff9444293cf3aa0832728bd P d2a651b924ca4e28b2b7f0b38c482057 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7dc9fb4c264b4a148b42a8ab1cfc2d7b (127.22.53.194:46701), d034c7e75f4f4b5dadc738c2527359fe (127.22.53.193:34085)
I20260926 07:34:03.407100 7787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:03.403387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.195:54009 (local address 127.22.53.193:34085)
0926 07:34:03.403542 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:03.403545 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:03.403557 (+ 12us) server_negotiation.cc:407] Connection header received
0926 07:34:03.403604 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:03.403607 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:03.403647 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:03.403736 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:34:03.404068 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:03.404836 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:03.406092 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:03.406504 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:03.406653 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:03.406712 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:03.406894 (+ 182us) server_negotiation.cc:299] Negotiation successful
0926 07:34:03.406972 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 07:34:03.407100 7775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:03.403156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.193:34085 (local address 127.22.53.195:54009)
0926 07:34:03.403271 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:03.403288 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:03.403392 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:03.403758 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:03.403761 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:03.403780 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:34:03.403916 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:03.403926 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:03.404976 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:03.404979 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:03.405897 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:03.405907 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:03.406650 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:03.406653 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:03.406767 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:03.406786 (+ 19us) client_negotiation.cc:769] Sending connection context
0926 07:34:03.406875 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 07:34:03.406957 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260926 07:34:03.407322 7785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:03.403431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.195:40701 (local address 127.22.53.194:46701)
0926 07:34:03.403580 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:03.403587 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:03.403940 (+ 353us) server_negotiation.cc:407] Connection header received
0926 07:34:03.404103 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:03.404107 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:03.404160 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:03.404233 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:34:03.404664 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:03.405228 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:03.406362 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:03.406769 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:03.406902 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:03.406951 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:03.407119 (+ 168us) server_negotiation.cc:299] Negotiation successful
0926 07:34:03.407203 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 07:34:03.407334 7799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:03.403162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.194:46701 (local address 127.22.53.195:40701)
0926 07:34:03.403826 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:03.403841 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:03.403978 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:03.404271 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:03.404277 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:03.404297 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:34:03.404524 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:03.404554 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:03.405351 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:03.405355 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:03.406244 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:03.406253 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:03.406895 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:03.406899 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:03.407000 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:03.407017 (+ 17us) client_negotiation.cc:769] Sending connection context
0926 07:34:03.407101 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 07:34:03.407186 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"mutex_wait_us":282,"thread_start_us":190,"threads_started":1}
I20260926 07:34:03.407896 7459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd8676c2dff9444293cf3aa0832728bd" candidate_uuid: "d2a651b924ca4e28b2b7f0b38c482057" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d034c7e75f4f4b5dadc738c2527359fe" is_pre_election: true
I20260926 07:34:03.408087 7593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd8676c2dff9444293cf3aa0832728bd" candidate_uuid: "d2a651b924ca4e28b2b7f0b38c482057" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7dc9fb4c264b4a148b42a8ab1cfc2d7b" is_pre_election: true
I20260926 07:34:03.408262 7459 raft_consensus.cc:2454] T dd8676c2dff9444293cf3aa0832728bd P d034c7e75f4f4b5dadc738c2527359fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2a651b924ca4e28b2b7f0b38c482057 in term 0.
I20260926 07:34:03.408406 7593 raft_consensus.cc:2454] T dd8676c2dff9444293cf3aa0832728bd P 7dc9fb4c264b4a148b42a8ab1cfc2d7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2a651b924ca4e28b2b7f0b38c482057 in term 0.
I20260926 07:34:03.408869 7663 leader_election.cc:302] T dd8676c2dff9444293cf3aa0832728bd P d2a651b924ca4e28b2b7f0b38c482057 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d034c7e75f4f4b5dadc738c2527359fe, d2a651b924ca4e28b2b7f0b38c482057; no voters:
I20260926 07:34:03.409139 7798 raft_consensus.cc:2781] T dd8676c2dff9444293cf3aa0832728bd P d2a651b924ca4e28b2b7f0b38c482057 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:34:03.409240 7798 raft_consensus.cc:482] T dd8676c2dff9444293cf3aa0832728bd P d2a651b924ca4e28b2b7f0b38c482057 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:34:03.409297 7798 raft_consensus.cc:3037] T dd8676c2dff9444293cf3aa0832728bd P d2a651b924ca4e28b2b7f0b38c482057 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:34:03.410431 7798 raft_consensus.cc:504] T dd8676c2dff9444293cf3aa0832728bd P d2a651b924ca4e28b2b7f0b38c482057 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7dc9fb4c264b4a148b42a8ab1cfc2d7b" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 46701 } } peers { permanent_uuid: "d034c7e75f4f4b5dadc738c2527359fe" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 34085 } } peers { permanent_uuid: "d2a651b924ca4e28b2b7f0b38c482057" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 45009 } }
I20260926 07:34:03.410866 7798 leader_election.cc:288] T dd8676c2dff9444293cf3aa0832728bd P d2a651b924ca4e28b2b7f0b38c482057 [CANDIDATE]: Term 1 election: Requested vote from peers 7dc9fb4c264b4a148b42a8ab1cfc2d7b (127.22.53.194:46701), d034c7e75f4f4b5dadc738c2527359fe (127.22.53.193:34085)
I20260926 07:34:03.411417 7593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd8676c2dff9444293cf3aa0832728bd" candidate_uuid: "d2a651b924ca4e28b2b7f0b38c482057" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7dc9fb4c264b4a148b42a8ab1cfc2d7b"
I20260926 07:34:03.411450 7459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd8676c2dff9444293cf3aa0832728bd" candidate_uuid: "d2a651b924ca4e28b2b7f0b38c482057" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d034c7e75f4f4b5dadc738c2527359fe"
I20260926 07:34:03.411576 7593 raft_consensus.cc:3037] T dd8676c2dff9444293cf3aa0832728bd P 7dc9fb4c264b4a148b42a8ab1cfc2d7b [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:34:03.411705 7459 raft_consensus.cc:3037] T dd8676c2dff9444293cf3aa0832728bd P d034c7e75f4f4b5dadc738c2527359fe [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:34:03.412879 7459 raft_consensus.cc:2454] T dd8676c2dff9444293cf3aa0832728bd P d034c7e75f4f4b5dadc738c2527359fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2a651b924ca4e28b2b7f0b38c482057 in term 1.
I20260926 07:34:03.412886 7593 raft_consensus.cc:2454] T dd8676c2dff9444293cf3aa0832728bd P 7dc9fb4c264b4a148b42a8ab1cfc2d7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2a651b924ca4e28b2b7f0b38c482057 in term 1.
I20260926 07:34:03.413273 7661 leader_election.cc:302] T dd8676c2dff9444293cf3aa0832728bd P d2a651b924ca4e28b2b7f0b38c482057 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7dc9fb4c264b4a148b42a8ab1cfc2d7b, d2a651b924ca4e28b2b7f0b38c482057; no voters:
I20260926 07:34:03.413458 7798 raft_consensus.cc:2781] T dd8676c2dff9444293cf3aa0832728bd P d2a651b924ca4e28b2b7f0b38c482057 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:34:03.413746 7798 raft_consensus.cc:686] T dd8676c2dff9444293cf3aa0832728bd P d2a651b924ca4e28b2b7f0b38c482057 [term 1 LEADER]: Becoming Leader. State: Replica: d2a651b924ca4e28b2b7f0b38c482057, State: Running, Role: LEADER
I20260926 07:34:03.414124 7798 consensus_queue.cc:237] T dd8676c2dff9444293cf3aa0832728bd P d2a651b924ca4e28b2b7f0b38c482057 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7dc9fb4c264b4a148b42a8ab1cfc2d7b" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 46701 } } peers { permanent_uuid: "d034c7e75f4f4b5dadc738c2527359fe" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 34085 } } peers { permanent_uuid: "d2a651b924ca4e28b2b7f0b38c482057" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 45009 } }
I20260926 07:34:03.417740 7312 catalog_manager.cc:5495] T dd8676c2dff9444293cf3aa0832728bd P d2a651b924ca4e28b2b7f0b38c482057 reported cstate change: term changed from 0 to 1, leader changed from <none> to d2a651b924ca4e28b2b7f0b38c482057 (127.22.53.195). New cstate: current_term: 1 leader_uuid: "d2a651b924ca4e28b2b7f0b38c482057" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7dc9fb4c264b4a148b42a8ab1cfc2d7b" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 46701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d034c7e75f4f4b5dadc738c2527359fe" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 34085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d2a651b924ca4e28b2b7f0b38c482057" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 45009 } health_report { overall_health: HEALTHY } } }
I20260926 07:34:03.442828 7789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:03.439243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38812 (local address 127.22.53.195:45009)
0926 07:34:03.439366 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:03.439369 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:03.439383 (+ 14us) server_negotiation.cc:407] Connection header received
0926 07:34:03.439468 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:03.439470 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:03.439529 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:03.439631 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:34:03.439825 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:03.440319 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:03.441178 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:03.441407 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:03.441483 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:03.441818 (+ 335us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:34:03.442271 (+ 453us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:34:03.442383 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:03.442563 (+ 180us) server_negotiation.cc:299] Negotiation successful
0926 07:34:03.442640 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 07:34:03.449069 7459 raft_consensus.cc:1261] T dd8676c2dff9444293cf3aa0832728bd P d034c7e75f4f4b5dadc738c2527359fe [term 1 FOLLOWER]: Refusing update from remote peer d2a651b924ca4e28b2b7f0b38c482057: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:34:03.449069 7593 raft_consensus.cc:1261] T dd8676c2dff9444293cf3aa0832728bd P 7dc9fb4c264b4a148b42a8ab1cfc2d7b [term 1 FOLLOWER]: Refusing update from remote peer d2a651b924ca4e28b2b7f0b38c482057: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:34:03.449970 7798 consensus_queue.cc:1035] T dd8676c2dff9444293cf3aa0832728bd P d2a651b924ca4e28b2b7f0b38c482057 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7dc9fb4c264b4a148b42a8ab1cfc2d7b" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 46701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:34:03.450248 7800 consensus_queue.cc:1035] T dd8676c2dff9444293cf3aa0832728bd P d2a651b924ca4e28b2b7f0b38c482057 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d034c7e75f4f4b5dadc738c2527359fe" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 34085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:34:03.457901 7803 mvcc.cc:205] Tried to move back new op lower bound from 7333511345959071744 to 7333511345827311616. Current Snapshot: MvccSnapshot[applied={T|T < 7333511345959071744}]
I20260926 07:34:03.458621 7805 mvcc.cc:205] Tried to move back new op lower bound from 7333511345959071744 to 7333511345827311616. Current Snapshot: MvccSnapshot[applied={T|T < 7333511345959071744}]
I20260926 07:34:03.460307 7804 mvcc.cc:205] Tried to move back new op lower bound from 7333511345959071744 to 7333511345827311616. Current Snapshot: MvccSnapshot[applied={T|T < 7333511345959071744}]
I20260926 07:34:03.468011 7312 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47688:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:34:03.468217 7312 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47688:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:34:03.471101 7312 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e168518ee12d4a5cb2ed5c24ed397d30: Sending DeleteTablet for 3 replicas of tablet dd8676c2dff9444293cf3aa0832728bd
I20260926 07:34:03.472065 7439 tablet_service.cc:1514] Processing DeleteTablet for tablet dd8676c2dff9444293cf3aa0832728bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:34:03 UTC) from {username='kudu', principal='kudu/127.22.53.254@KRBTEST.COM'} at 127.0.0.1:41328
I20260926 07:34:03.472147 7573 tablet_service.cc:1514] Processing DeleteTablet for tablet dd8676c2dff9444293cf3aa0832728bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:34:03 UTC) from {username='kudu', principal='kudu/127.22.53.254@KRBTEST.COM'} at 127.0.0.1:38960
I20260926 07:34:03.472352 7707 tablet_service.cc:1514] Processing DeleteTablet for tablet dd8676c2dff9444293cf3aa0832728bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:34:03 UTC) from {username='kudu', principal='kudu/127.22.53.254@KRBTEST.COM'} at 127.0.0.1:38796
I20260926 07:34:03.472744 7812 tablet_replica.cc:330] T dd8676c2dff9444293cf3aa0832728bd P d034c7e75f4f4b5dadc738c2527359fe: stopping tablet replica
I20260926 07:34:03.473081 7812 raft_consensus.cc:2229] T dd8676c2dff9444293cf3aa0832728bd P d034c7e75f4f4b5dadc738c2527359fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:34:03.473449 7812 raft_consensus.cc:2258] T dd8676c2dff9444293cf3aa0832728bd P d034c7e75f4f4b5dadc738c2527359fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:34:03.473850 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 7376
I20260926 07:34:03.475188 7813 tablet_replica.cc:330] T dd8676c2dff9444293cf3aa0832728bd P 7dc9fb4c264b4a148b42a8ab1cfc2d7b: stopping tablet replica
I20260926 07:34:03.475517 7813 raft_consensus.cc:2229] T dd8676c2dff9444293cf3aa0832728bd P 7dc9fb4c264b4a148b42a8ab1cfc2d7b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:34:03.475821 7813 raft_consensus.cc:2258] T dd8676c2dff9444293cf3aa0832728bd P 7dc9fb4c264b4a148b42a8ab1cfc2d7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:34:03.477535 7813 ts_tablet_manager.cc:1905] T dd8676c2dff9444293cf3aa0832728bd P 7dc9fb4c264b4a148b42a8ab1cfc2d7b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:34:03.480443 7813 ts_tablet_manager.cc:1918] T dd8676c2dff9444293cf3aa0832728bd P 7dc9fb4c264b4a148b42a8ab1cfc2d7b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:34:03.480585 7813 log.cc:1196] T dd8676c2dff9444293cf3aa0832728bd P 7dc9fb4c264b4a148b42a8ab1cfc2d7b: Deleting WAL directory at /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407783529717-22743-0/minicluster-data/ts-1/wal/wals/dd8676c2dff9444293cf3aa0832728bd
I20260926 07:34:03.480881 7813 ts_tablet_manager.cc:1939] T dd8676c2dff9444293cf3aa0832728bd P 7dc9fb4c264b4a148b42a8ab1cfc2d7b: Deleting consensus metadata
I20260926 07:34:03.482366 7814 tablet_replica.cc:330] T dd8676c2dff9444293cf3aa0832728bd P d2a651b924ca4e28b2b7f0b38c482057: stopping tablet replica
I20260926 07:34:03.482389 7299 catalog_manager.cc:4841] TS 7dc9fb4c264b4a148b42a8ab1cfc2d7b (127.22.53.194:46701): tablet dd8676c2dff9444293cf3aa0832728bd (table test-table [id=6e8d87d279c74fe2a34e8942845311aa]) successfully deleted
I20260926 07:34:03.482760 7814 raft_consensus.cc:2229] T dd8676c2dff9444293cf3aa0832728bd P d2a651b924ca4e28b2b7f0b38c482057 [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:34:03.483237 7814 raft_consensus.cc:2258] T dd8676c2dff9444293cf3aa0832728bd P d2a651b924ca4e28b2b7f0b38c482057 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:34:03.485061 7814 ts_tablet_manager.cc:1905] T dd8676c2dff9444293cf3aa0832728bd P d2a651b924ca4e28b2b7f0b38c482057: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 07:34:03.485414 7301 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.53.193:34085 (error 108)
I20260926 07:34:03.487000 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 7510
I20260926 07:34:03.487030 7786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:03.486711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.193:34085 (local address 127.0.0.1:41342)
0926 07:34:03.486845 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:03.486927 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.193:34085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260926 07:34:03.487891 7814 ts_tablet_manager.cc:1918] T dd8676c2dff9444293cf3aa0832728bd P d2a651b924ca4e28b2b7f0b38c482057: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:34:03.487993 7814 log.cc:1196] T dd8676c2dff9444293cf3aa0832728bd P d2a651b924ca4e28b2b7f0b38c482057: Deleting WAL directory at /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407783529717-22743-0/minicluster-data/ts-2/wal/wals/dd8676c2dff9444293cf3aa0832728bd
I20260926 07:34:03.488318 7814 ts_tablet_manager.cc:1939] T dd8676c2dff9444293cf3aa0832728bd P d2a651b924ca4e28b2b7f0b38c482057: Deleting consensus metadata
I20260926 07:34:03.489730 7298 catalog_manager.cc:4841] TS d2a651b924ca4e28b2b7f0b38c482057 (127.22.53.195:45009): tablet dd8676c2dff9444293cf3aa0832728bd (table test-table [id=6e8d87d279c74fe2a34e8942845311aa]) successfully deleted
W20260926 07:34:03.490074 7301 catalog_manager.cc:4584] TS d034c7e75f4f4b5dadc738c2527359fe (127.22.53.193:34085): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dd8676c2dff9444293cf3aa0832728bd: Network error: Client connection negotiation failed: client connection to 127.22.53.193:34085: connect: Connection refused (error 111)
I20260926 07:34:03.494752 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 7644
I20260926 07:34:03.501099 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 7282
2026-09-26T07:34:03Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2420 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-26T07:34:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:34:03Z Disabled control of system clock
I20260926 07:34:03.532720 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407783529717-22743-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407783529717-22743-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407783529717-22743-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.53.254:33571
--webserver_interface=127.22.53.254
--webserver_port=0
--builtin_ntp_servers=127.22.53.212:33547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.53.254:33571
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 07:34:03.648514 7819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:34:03.648819 7819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:34:03.648857 7819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:34:03.652927 7819 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:34:03.652989 7819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:34:03.653004 7819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:34:03.653021 7819 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:34:03.653036 7819 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:34:03.657929 7819 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:33547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407783529717-22743-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407783529717-22743-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.53.254:33571
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.53.254:33571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.53.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7819
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407783529717-22743-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:34:03.659145 7819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:34:03.660498 7819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:34:03.669317 7827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:03.669306 7825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:03.669306 7824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:34:03.669708 7819 server_base.cc:1034] running on GCE node
I20260926 07:34:03.670393 7819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:34:03.671700 7819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:34:03.672899 7819 hybrid_clock.cc:648] HybridClock initialized: now 1790408043672873 us; error 42 us; skew 500 ppm
I20260926 07:34:03.674860 7819 webserver.cc:460] Webserver started at http://127.22.53.254:44369/ using document root <none> and password file <none>
I20260926 07:34:03.675462 7819 fs_manager.cc:359] Metadata directory not provided
I20260926 07:34:03.675557 7819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:34:03.675794 7819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:34:03.678048 7819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407783529717-22743-0/minicluster-data/master-0/data/instance:
uuid: "74b215d152f34aac954fc55e3f2850fb"
format_stamp: "Formatted at 2026-09-26 07:34:03 on dist-test-slave-wdjc"
server_key: "f97e24575875c3e1ec54d7a9a495789d"
server_key_iv: "54aedc41f9aeac16bfc76e136a9fab64"
server_key_version: "encryptionkey@0"
I20260926 07:34:03.678686 7819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407783529717-22743-0/minicluster-data/master-0/wal/instance:
uuid: "74b215d152f34aac954fc55e3f2850fb"
format_stamp: "Formatted at 2026-09-26 07:34:03 on dist-test-slave-wdjc"
server_key: "f97e24575875c3e1ec54d7a9a495789d"
server_key_iv: "54aedc41f9aeac16bfc76e136a9fab64"
server_key_version: "encryptionkey@0"
I20260926 07:34:03.682819 7819 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 07:34:03.685680 7833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:03.687202 7819 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:34:03.687379 7819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407783529717-22743-0/minicluster-data/master-0/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407783529717-22743-0/minicluster-data/master-0/wal
uuid: "74b215d152f34aac954fc55e3f2850fb"
format_stamp: "Formatted at 2026-09-26 07:34:03 on dist-test-slave-wdjc"
server_key: "f97e24575875c3e1ec54d7a9a495789d"
server_key_iv: "54aedc41f9aeac16bfc76e136a9fab64"
server_key_version: "encryptionkey@0"
I20260926 07:34:03.687522 7819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407783529717-22743-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407783529717-22743-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407783529717-22743-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:34:03.742195 7819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:34:03.743436 7819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:34:03.743690 7819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:34:03.751180 7819 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.254:33571
I20260926 07:34:03.751214 7885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.254:33571 every 8 connection(s)
I20260926 07:34:03.752409 7819 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
I20260926 07:34:03.755816 7886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:34:03.760123 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 7819
I20260926 07:34:03.760326 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407783529717-22743-0/minicluster-data/master-0/wal/instance
I20260926 07:34:03.760697 22743 external_mini_cluster.cc:1428] Setting key d3540e7d725fe9cbc67efd838ebf52b7
I20260926 07:34:03.762533 7886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74b215d152f34aac954fc55e3f2850fb: Bootstrap starting.
I20260926 07:34:03.765468 7886 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 74b215d152f34aac954fc55e3f2850fb: Neither blocks nor log segments found. Creating new log.
I20260926 07:34:03.766435 7886 log.cc:824] T 00000000000000000000000000000000 P 74b215d152f34aac954fc55e3f2850fb: Log is configured to *not* fsync() on all Append() calls
I20260926 07:34:03.768805 7886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74b215d152f34aac954fc55e3f2850fb: No bootstrap required, opened a new log
I20260926 07:34:03.772305 7889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:03.762354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39356 (local address 127.22.53.254:33571)
0926 07:34:03.763150 (+ 796us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:03.763171 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:03.763215 (+ 44us) server_negotiation.cc:407] Connection header received
0926 07:34:03.764147 (+ 932us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:03.764171 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:03.764718 (+ 547us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:03.765132 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:03.765618 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:03.767072 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:03.768062 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:03.768766 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:03.768948 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:03.769888 (+ 940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:03.769894 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:03.769912 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:34:03.770445 (+ 533us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:03.770522 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:34:03.770550 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:03.770648 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:03.770764 (+ 116us) server_negotiation.cc:299] Negotiation successful
0926 07:34:03.771072 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":174,"threads_started":1}
I20260926 07:34:03.773090 7886 raft_consensus.cc:348] T 00000000000000000000000000000000 P 74b215d152f34aac954fc55e3f2850fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74b215d152f34aac954fc55e3f2850fb" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 33571 } }
I20260926 07:34:03.773370 7886 raft_consensus.cc:374] T 00000000000000000000000000000000 P 74b215d152f34aac954fc55e3f2850fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:34:03.773449 7886 raft_consensus.cc:729] T 00000000000000000000000000000000 P 74b215d152f34aac954fc55e3f2850fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74b215d152f34aac954fc55e3f2850fb, State: Initialized, Role: FOLLOWER
I20260926 07:34:03.773962 7886 consensus_queue.cc:260] T 00000000000000000000000000000000 P 74b215d152f34aac954fc55e3f2850fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74b215d152f34aac954fc55e3f2850fb" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 33571 } }
I20260926 07:34:03.774130 7886 raft_consensus.cc:388] T 00000000000000000000000000000000 P 74b215d152f34aac954fc55e3f2850fb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:34:03.774212 7886 raft_consensus.cc:482] T 00000000000000000000000000000000 P 74b215d152f34aac954fc55e3f2850fb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:34:03.774318 7886 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 74b215d152f34aac954fc55e3f2850fb [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:34:03.775557 7886 raft_consensus.cc:504] T 00000000000000000000000000000000 P 74b215d152f34aac954fc55e3f2850fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74b215d152f34aac954fc55e3f2850fb" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 33571 } }
I20260926 07:34:03.775938 7886 leader_election.cc:302] T 00000000000000000000000000000000 P 74b215d152f34aac954fc55e3f2850fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 74b215d152f34aac954fc55e3f2850fb; no voters:
I20260926 07:34:03.776304 7886 leader_election.cc:288] T 00000000000000000000000000000000 P 74b215d152f34aac954fc55e3f2850fb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:34:03.776432 7891 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 74b215d152f34aac954fc55e3f2850fb [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:34:03.776718 7891 raft_consensus.cc:686] T 00000000000000000000000000000000 P 74b215d152f34aac954fc55e3f2850fb [term 1 LEADER]: Becoming Leader. State: Replica: 74b215d152f34aac954fc55e3f2850fb, State: Running, Role: LEADER
I20260926 07:34:03.777043 7891 consensus_queue.cc:237] T 00000000000000000000000000000000 P 74b215d152f34aac954fc55e3f2850fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74b215d152f34aac954fc55e3f2850fb" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 33571 } }
I20260926 07:34:03.777711 7886 sys_catalog.cc:564] T 00000000000000000000000000000000 P 74b215d152f34aac954fc55e3f2850fb [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:34:03.778748 7893 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74b215d152f34aac954fc55e3f2850fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 74b215d152f34aac954fc55e3f2850fb. Latest consensus state: current_term: 1 leader_uuid: "74b215d152f34aac954fc55e3f2850fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74b215d152f34aac954fc55e3f2850fb" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 33571 } } }
I20260926 07:34:03.778885 7893 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74b215d152f34aac954fc55e3f2850fb [sys.catalog]: This master's current role is: LEADER
I20260926 07:34:03.779215 7892 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74b215d152f34aac954fc55e3f2850fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "74b215d152f34aac954fc55e3f2850fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74b215d152f34aac954fc55e3f2850fb" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 33571 } } }
I20260926 07:34:03.779312 7892 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74b215d152f34aac954fc55e3f2850fb [sys.catalog]: This master's current role is: LEADER
I20260926 07:34:03.779577 7900 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:34:03.783779 7900 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:34:03.790365 7900 catalog_manager.cc:1347] Generated new cluster ID: 7bfa963396c647738d60fecf9bbd4c6f
I20260926 07:34:03.790472 7900 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:34:03.798512 7900 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:34:03.799877 7900 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:34:03.812014 7900 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 74b215d152f34aac954fc55e3f2850fb: Generated new TSK 0
I20260926 07:34:03.812996 7900 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:34:03.823459 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407783529717-22743-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407783529717-22743-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407783529717-22743-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407783529717-22743-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.53.193:0
--local_ip_for_outbound_sockets=127.22.53.193
--webserver_interface=127.22.53.193
--webserver_port=0
--tserver_master_addrs=127.22.53.254:33571
--builtin_ntp_servers=127.22.53.212:33547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 07:34:03.942528 7910 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:34:03.942879 7910 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:34:03.942925 7910 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:34:03.947487 7910 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:34:03.947589 7910 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:34:03.947654 7910 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.53.193
I20260926 07:34:03.952939 7910 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:33547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407783529717-22743-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407783529717-22743-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.53.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407783529717-22743-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.53.193
--webserver_port=0
--tserver_master_addrs=127.22.53.254:33571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7910
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.53.193
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407783529717-22743-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:34:03.954245 7910 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:34:03.955819 7910 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:34:03.966507 7918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:03.966549 7915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:03.966544 7916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:34:03.966969 7910 server_base.cc:1034] running on GCE node
I20260926 07:34:03.967548 7910 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:34:03.968326 7910 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:34:03.969583 7910 hybrid_clock.cc:648] HybridClock initialized: now 1790408043969575 us; error 62 us; skew 500 ppm
I20260926 07:34:03.971818 7910 webserver.cc:460] Webserver started at http://127.22.53.193:35113/ using document root <none> and password file <none>
I20260926 07:34:03.972519 7910 fs_manager.cc:359] Metadata directory not provided
I20260926 07:34:03.972681 7910 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:34:03.972949 7910 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:34:03.975389 7910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407783529717-22743-0/minicluster-data/ts-0/data/instance:
uuid: "3b9478c5d0c54ea2bcd246ce7d2db2c6"
format_stamp: "Formatted at 2026-09-26 07:34:03 on dist-test-slave-wdjc"
server_key: "d349971e6db4e0c2d97d5391c4040e8a"
server_key_iv: "a989928d43603eb3e78afa0040459085"
server_key_version: "encryptionkey@0"
I20260926 07:34:03.976110 7910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407783529717-22743-0/minicluster-data/ts-0/wal/instance:
uuid: "3b9478c5d0c54ea2bcd246ce7d2db2c6"
format_stamp: "Formatted at 2026-09-26 07:34:03 on dist-test-slave-wdjc"
server_key: "d349971e6db4e0c2d97d5391c4040e8a"
server_key_iv: "a989928d43603eb3e78afa0040459085"
server_key_version: "encryptionkey@0"
I20260926 07:34:03.981061 7910 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260926 07:34:03.984242 7924 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:03.986094 7910 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 07:34:03.986331 7910 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407783529717-22743-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407783529717-22743-0/minicluster-data/ts-0/wal
uuid: "3b9478c5d0c54ea2bcd246ce7d2db2c6"
format_stamp: "Formatted at 2026-09-26 07:34:03 on dist-test-slave-wdjc"
server_key: "d349971e6db4e0c2d97d5391c4040e8a"
server_key_iv: "a989928d43603eb3e78afa0040459085"
server_key_version: "encryptionkey@0"
I20260926 07:34:03.986492 7910 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407783529717-22743-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407783529717-22743-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407783529717-22743-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:34:04.010809 7910 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:34:04.011579 7910 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:34:04.011804 7910 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:34:04.012444 7910 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:34:04.013469 7910 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:34:04.013540 7910 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:04.013617 7910 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:34:04.013662 7910 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:04.023537 7910 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.193:38253
I20260926 07:34:04.023566 8037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.193:38253 every 8 connection(s)
I20260926 07:34:04.024526 7910 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407783529717-22743-0/minicluster-data/ts-0/data/info.pb
I20260926 07:34:04.030938 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 7910
I20260926 07:34:04.031106 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407783529717-22743-0/minicluster-data/ts-0/wal/instance
I20260926 07:34:04.031445 22743 external_mini_cluster.cc:1428] Setting key f963bd34479ecae8f35779bbee2e24a0
I20260926 07:34:04.033566 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407783529717-22743-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407783529717-22743-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407783529717-22743-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407783529717-22743-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.53.194:0
--local_ip_for_outbound_sockets=127.22.53.194
--webserver_interface=127.22.53.194
--webserver_port=0
--tserver_master_addrs=127.22.53.254:33571
--builtin_ntp_servers=127.22.53.212:33547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 07:34:04.036023 7889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:04.026743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.193:55313 (local address 127.22.53.254:33571)
0926 07:34:04.026927 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:04.026932 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:04.028154 (+ 1222us) server_negotiation.cc:407] Connection header received
0926 07:34:04.030141 (+ 1987us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:04.030144 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:04.030204 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:04.030288 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:04.031457 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:04.032131 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:04.033548 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:04.033895 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:04.034013 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:04.035434 (+ 1421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:04.035437 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:04.035440 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:34:04.035454 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:04.035478 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:34:04.035486 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:04.035556 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:04.035781 (+ 225us) server_negotiation.cc:299] Negotiation successful
0926 07:34:04.035868 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 07:34:04.037113 8040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:04.027227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:33571 (local address 127.22.53.193:55313)
0926 07:34:04.027975 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:04.028021 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:04.029867 (+ 1846us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:04.030488 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:04.030495 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:04.030595 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:04.031280 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:04.031298 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:04.032284 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:04.032288 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:04.033417 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:04.033429 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:04.034067 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:04.034072 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:04.034256 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:04.035015 (+ 759us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:34:04.035018 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:04.035095 (+ 77us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:34:04.035239 (+ 144us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:34:04.035580 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:34:04.035590 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:34:04.035596 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 07:34:04.035751 (+ 155us) client_negotiation.cc:240] Negotiation successful
0926 07:34:04.036143 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":133,"threads_started":1}
I20260926 07:34:04.038542 8038 heartbeater.cc:347] Connected to a master server at 127.22.53.254:33571
I20260926 07:34:04.038815 8038 heartbeater.cc:464] Registering TS with master...
I20260926 07:34:04.039510 8038 heartbeater.cc:511] Master 127.22.53.254:33571 requested a full tablet report, sending...
I20260926 07:34:04.041603 7850 ts_manager.cc:194] Registered new tserver with Master: 3b9478c5d0c54ea2bcd246ce7d2db2c6 (127.22.53.193:38253)
I20260926 07:34:04.043592 7850 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.53.193:55313
W20260926 07:34:04.150543 8041 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:34:04.150849 8041 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:34:04.150892 8041 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:34:04.155112 8041 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:34:04.155203 8041 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:34:04.155263 8041 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.53.194
I20260926 07:34:04.160465 8041 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:33547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407783529717-22743-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407783529717-22743-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.53.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407783529717-22743-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.53.194
--webserver_port=0
--tserver_master_addrs=127.22.53.254:33571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8041
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.53.194
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407783529717-22743-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:34:04.161659 8041 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:34:04.163080 8041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:34:04.173100 8047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:04.173100 8046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:04.173158 8049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:34:04.173380 8041 server_base.cc:1034] running on GCE node
I20260926 07:34:04.173878 8041 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:34:04.174598 8041 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:34:04.175808 8041 hybrid_clock.cc:648] HybridClock initialized: now 1790408044175773 us; error 50 us; skew 500 ppm
I20260926 07:34:04.178086 8041 webserver.cc:460] Webserver started at http://127.22.53.194:43231/ using document root <none> and password file <none>
I20260926 07:34:04.178820 8041 fs_manager.cc:359] Metadata directory not provided
I20260926 07:34:04.178915 8041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:34:04.179140 8041 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:34:04.181447 8041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407783529717-22743-0/minicluster-data/ts-1/data/instance:
uuid: "e113b78131c94dae946ba53eb43d5f36"
format_stamp: "Formatted at 2026-09-26 07:34:04 on dist-test-slave-wdjc"
server_key: "50eed33e6a60dc3c62fcd073ea39045a"
server_key_iv: "96eddee421fbc0acfddc080e4305192a"
server_key_version: "encryptionkey@0"
I20260926 07:34:04.182037 8041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407783529717-22743-0/minicluster-data/ts-1/wal/instance:
uuid: "e113b78131c94dae946ba53eb43d5f36"
format_stamp: "Formatted at 2026-09-26 07:34:04 on dist-test-slave-wdjc"
server_key: "50eed33e6a60dc3c62fcd073ea39045a"
server_key_iv: "96eddee421fbc0acfddc080e4305192a"
server_key_version: "encryptionkey@0"
I20260926 07:34:04.186372 8041 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 07:34:04.190246 8055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:04.193240 8041 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20260926 07:34:04.193425 8041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407783529717-22743-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407783529717-22743-0/minicluster-data/ts-1/wal
uuid: "e113b78131c94dae946ba53eb43d5f36"
format_stamp: "Formatted at 2026-09-26 07:34:04 on dist-test-slave-wdjc"
server_key: "50eed33e6a60dc3c62fcd073ea39045a"
server_key_iv: "96eddee421fbc0acfddc080e4305192a"
server_key_version: "encryptionkey@0"
I20260926 07:34:04.193638 8041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407783529717-22743-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407783529717-22743-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407783529717-22743-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:34:04.204366 8041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:34:04.205183 8041 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:34:04.205395 8041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:34:04.206022 8041 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:34:04.207041 8041 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:34:04.207089 8041 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:04.207132 8041 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:34:04.207185 8041 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:04.216581 8041 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.194:39255
I20260926 07:34:04.216682 8168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.194:39255 every 8 connection(s)
I20260926 07:34:04.217621 8041 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407783529717-22743-0/minicluster-data/ts-1/data/info.pb
I20260926 07:34:04.221443 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 8041
I20260926 07:34:04.221554 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407783529717-22743-0/minicluster-data/ts-1/wal/instance
I20260926 07:34:04.221864 22743 external_mini_cluster.cc:1428] Setting key 7ac4f914404af61648d6fa59c0132e70
I20260926 07:34:04.224274 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407783529717-22743-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407783529717-22743-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407783529717-22743-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407783529717-22743-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.53.195:0
--local_ip_for_outbound_sockets=127.22.53.195
--webserver_interface=127.22.53.195
--webserver_port=0
--tserver_master_addrs=127.22.53.254:33571
--builtin_ntp_servers=127.22.53.212:33547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 07:34:04.230392 7889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:04.219595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.194:33485 (local address 127.22.53.254:33571)
0926 07:34:04.219779 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:04.219784 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:04.220940 (+ 1156us) server_negotiation.cc:407] Connection header received
0926 07:34:04.222598 (+ 1658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:04.222601 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:04.222669 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:04.222778 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:04.224350 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:04.225581 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:04.227621 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:04.227998 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:04.228123 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:04.229771 (+ 1648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:04.229775 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:04.229778 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:34:04.229796 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:04.229823 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:34:04.229832 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:04.229914 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:04.230116 (+ 202us) server_negotiation.cc:299] Negotiation successful
0926 07:34:04.230204 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 07:34:04.231734 8171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:04.220036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:33571 (local address 127.22.53.194:33485)
0926 07:34:04.220785 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:04.220830 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:04.222352 (+ 1522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:04.223143 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:04.223155 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:04.223304 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:04.224121 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:04.224145 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:04.225776 (+ 1631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:04.225782 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:04.227458 (+ 1676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:04.227475 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:04.228211 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:04.228218 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:04.228465 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:04.229337 (+ 872us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:34:04.229339 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:04.229419 (+ 80us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:34:04.229539 (+ 120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:34:04.229929 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:34:04.229939 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:34:04.229945 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 07:34:04.230653 (+ 708us) client_negotiation.cc:240] Negotiation successful
0926 07:34:04.230928 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":128,"threads_started":1}
I20260926 07:34:04.233302 8169 heartbeater.cc:347] Connected to a master server at 127.22.53.254:33571
I20260926 07:34:04.233608 8169 heartbeater.cc:464] Registering TS with master...
I20260926 07:34:04.234190 8169 heartbeater.cc:511] Master 127.22.53.254:33571 requested a full tablet report, sending...
I20260926 07:34:04.235453 7850 ts_manager.cc:194] Registered new tserver with Master: e113b78131c94dae946ba53eb43d5f36 (127.22.53.194:39255)
I20260926 07:34:04.236367 7850 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.53.194:33485
W20260926 07:34:04.343236 8172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:34:04.343544 8172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:34:04.343585 8172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:34:04.348299 8172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:34:04.348398 8172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:34:04.348462 8172 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.53.195
I20260926 07:34:04.353832 8172 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:33547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407783529717-22743-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407783529717-22743-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.53.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407783529717-22743-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.53.195
--webserver_port=0
--tserver_master_addrs=127.22.53.254:33571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8172
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.53.195
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407783529717-22743-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:34:04.355011 8172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:34:04.356844 8172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:34:04.368137 8178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:34:04.368203 8172 server_base.cc:1034] running on GCE node
W20260926 07:34:04.368175 8177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:04.368448 8180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:34:04.369006 8172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:34:04.369750 8172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:34:04.370934 8172 hybrid_clock.cc:648] HybridClock initialized: now 1790408044370903 us; error 44 us; skew 500 ppm
I20260926 07:34:04.373258 8172 webserver.cc:460] Webserver started at http://127.22.53.195:45645/ using document root <none> and password file <none>
I20260926 07:34:04.373891 8172 fs_manager.cc:359] Metadata directory not provided
I20260926 07:34:04.373962 8172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:34:04.374230 8172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:34:04.376500 8172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407783529717-22743-0/minicluster-data/ts-2/data/instance:
uuid: "419671ebd8524b7aac4c3551a44a3e2a"
format_stamp: "Formatted at 2026-09-26 07:34:04 on dist-test-slave-wdjc"
server_key: "4cfd2b2f950c3ea2dbe8594565f1c961"
server_key_iv: "11d824a9aa346dc573666a80e1f1cbc3"
server_key_version: "encryptionkey@0"
I20260926 07:34:04.377233 8172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407783529717-22743-0/minicluster-data/ts-2/wal/instance:
uuid: "419671ebd8524b7aac4c3551a44a3e2a"
format_stamp: "Formatted at 2026-09-26 07:34:04 on dist-test-slave-wdjc"
server_key: "4cfd2b2f950c3ea2dbe8594565f1c961"
server_key_iv: "11d824a9aa346dc573666a80e1f1cbc3"
server_key_version: "encryptionkey@0"
I20260926 07:34:04.381579 8172 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 07:34:04.384392 8186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:04.385866 8172 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:34:04.386023 8172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407783529717-22743-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407783529717-22743-0/minicluster-data/ts-2/wal
uuid: "419671ebd8524b7aac4c3551a44a3e2a"
format_stamp: "Formatted at 2026-09-26 07:34:04 on dist-test-slave-wdjc"
server_key: "4cfd2b2f950c3ea2dbe8594565f1c961"
server_key_iv: "11d824a9aa346dc573666a80e1f1cbc3"
server_key_version: "encryptionkey@0"
I20260926 07:34:04.386157 8172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407783529717-22743-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407783529717-22743-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407783529717-22743-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:34:04.401027 8172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:34:04.401851 8172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:34:04.402069 8172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:34:04.402721 8172 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:34:04.403739 8172 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:34:04.403790 8172 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:04.403855 8172 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:34:04.403889 8172 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:04.413645 8172 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.195:40023
I20260926 07:34:04.413671 8299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.195:40023 every 8 connection(s)
I20260926 07:34:04.414637 8172 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407783529717-22743-0/minicluster-data/ts-2/data/info.pb
I20260926 07:34:04.423573 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 8172
I20260926 07:34:04.423714 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407783529717-22743-0/minicluster-data/ts-2/wal/instance
I20260926 07:34:04.424009 22743 external_mini_cluster.cc:1428] Setting key 66d70105bf261488f1c2736f4fdbe34b
I20260926 07:34:04.426225 7889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:04.416611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.195:44987 (local address 127.22.53.254:33571)
0926 07:34:04.416787 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:04.416792 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:04.418033 (+ 1241us) server_negotiation.cc:407] Connection header received
0926 07:34:04.419505 (+ 1472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:04.419508 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:04.419565 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:04.419713 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:04.421063 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:04.421943 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:04.423771 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:04.424101 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:04.424210 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:04.425701 (+ 1491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:04.425705 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:04.425708 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:34:04.425722 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:04.425758 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:34:04.425769 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:04.425847 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:04.426004 (+ 157us) server_negotiation.cc:299] Negotiation successful
0926 07:34:04.426075 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260926 07:34:04.427179 8302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:04.417090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:33571 (local address 127.22.53.195:44987)
0926 07:34:04.417846 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:04.417895 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:04.419264 (+ 1369us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:04.419940 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:04.419948 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:04.420042 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:04.420856 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:04.420880 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:04.422098 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:04.422102 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:04.423591 (+ 1489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:04.423605 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:04.424274 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:04.424279 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:04.424512 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:04.425327 (+ 815us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:34:04.425329 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:04.425398 (+ 69us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:34:04.425503 (+ 105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:34:04.425857 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:34:04.425863 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:34:04.425868 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 07:34:04.425964 (+ 96us) client_negotiation.cc:240] Negotiation successful
0926 07:34:04.426236 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":127,"threads_started":1}
I20260926 07:34:04.428393 8300 heartbeater.cc:347] Connected to a master server at 127.22.53.254:33571
I20260926 07:34:04.428681 8300 heartbeater.cc:464] Registering TS with master...
I20260926 07:34:04.429272 8300 heartbeater.cc:511] Master 127.22.53.254:33571 requested a full tablet report, sending...
I20260926 07:34:04.430444 7850 ts_manager.cc:194] Registered new tserver with Master: 419671ebd8524b7aac4c3551a44a3e2a (127.22.53.195:40023)
I20260926 07:34:04.431088 7850 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.53.195:44987
I20260926 07:34:04.438469 22743 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:34:04.445123 7889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:04.440590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39366 (local address 127.22.53.254:33571)
0926 07:34:04.440758 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:04.440763 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:04.440900 (+ 137us) server_negotiation.cc:407] Connection header received
0926 07:34:04.441073 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:04.441076 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:04.441145 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:04.441231 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:04.441524 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:04.442173 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:04.443087 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:04.443316 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:04.443425 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:04.444644 (+ 1219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:04.444648 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:04.444650 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:34:04.444664 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:04.444685 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:34:04.444694 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:04.444773 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:04.444889 (+ 116us) server_negotiation.cc:299] Negotiation successful
0926 07:34:04.444959 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 07:34:04.448352 7850 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39366:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:34:04.451571 7850 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:34:04.467022 8312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:04.460816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.193:38253 (local address 127.0.0.1:53330)
0926 07:34:04.461266 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:04.461317 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:04.461446 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:04.462342 (+ 896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:04.462349 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:04.462378 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:34:04.462543 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:04.462552 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:04.464195 (+ 1643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:04.464198 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:04.465348 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:04.465358 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:04.466416 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:04.466420 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:04.466543 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:04.466582 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 07:34:04.466694 (+ 112us) client_negotiation.cc:240] Negotiation successful
0926 07:34:04.466844 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"mutex_wait_us":37,"thread_start_us":116,"threads_started":1}
I20260926 07:34:04.467028 8313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:04.460952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53330 (local address 127.22.53.193:38253)
0926 07:34:04.461777 (+ 825us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:04.461790 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:04.461816 (+ 26us) server_negotiation.cc:407] Connection header received
0926 07:34:04.461924 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:04.461930 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:04.462147 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:04.462711 (+ 564us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:34:04.462846 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:04.464070 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:04.465501 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:04.466275 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:04.466415 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:04.466636 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:04.466747 (+ 111us) server_negotiation.cc:299] Negotiation successful
0926 07:34:04.466857 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":712,"thread_start_us":106,"threads_started":1}
I20260926 07:34:04.467546 8315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:04.460854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.194:39255 (local address 127.0.0.1:41494)
0926 07:34:04.461922 (+ 1068us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:04.461939 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:04.462056 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:04.462692 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:04.462696 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:04.462729 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:34:04.462839 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:04.462846 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:04.463966 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:04.463973 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:04.465547 (+ 1574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:04.465559 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:04.466905 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:04.466909 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:04.467017 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:04.467056 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 07:34:04.467304 (+ 248us) client_negotiation.cc:240] Negotiation successful
0926 07:34:04.467391 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1013,"mutex_wait_us":88,"thread_start_us":114,"threads_started":1}
I20260926 07:34:04.467862 8316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:04.461962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37124 (local address 127.22.53.195:40023)
0926 07:34:04.462262 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:04.462272 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:04.462674 (+ 402us) server_negotiation.cc:407] Connection header received
0926 07:34:04.462754 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:04.462758 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:04.462931 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:04.463075 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:34:04.463286 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:04.464330 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:04.465878 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:04.466980 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:04.467093 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:04.467315 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:04.467593 (+ 278us) server_negotiation.cc:299] Negotiation successful
0926 07:34:04.467700 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":103,"threads_started":1}
I20260926 07:34:04.468021 8311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:04.461599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.195:40023 (local address 127.0.0.1:37124)
0926 07:34:04.462479 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:04.462493 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:04.462572 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:04.463074 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:04.463077 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:04.463096 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:34:04.463176 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:04.463181 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:04.464444 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:04.464450 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:04.465752 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:04.465759 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:04.467371 (+ 1612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:04.467374 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:04.467452 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:04.467474 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 07:34:04.467551 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 07:34:04.467808 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":822,"thread_start_us":122,"threads_started":1}
I20260926 07:34:04.469013 8314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:04.461475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41494 (local address 127.22.53.194:39255)
0926 07:34:04.461811 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:04.461824 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:04.462032 (+ 208us) server_negotiation.cc:407] Connection header received
0926 07:34:04.462210 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:04.462215 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:04.462390 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:04.462539 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:34:04.462973 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:04.463850 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:04.465700 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:04.466550 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:04.466658 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:04.466838 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:04.468274 (+ 1436us) server_negotiation.cc:299] Negotiation successful
0926 07:34:04.468827 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":132,"threads_started":1}
I20260926 07:34:04.470718 7972 tablet_service.cc:1467] Processing CreateTablet for tablet 45ca50ef1b5b47d0a7405f735161210e (DEFAULT_TABLE table=test-table [id=15949659fc7a4af18af37f24aba96018]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:34:04.471698 8234 tablet_service.cc:1467] Processing CreateTablet for tablet 45ca50ef1b5b47d0a7405f735161210e (DEFAULT_TABLE table=test-table [id=15949659fc7a4af18af37f24aba96018]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:34:04.472000 7972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45ca50ef1b5b47d0a7405f735161210e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:34:04.471864 8103 tablet_service.cc:1467] Processing CreateTablet for tablet 45ca50ef1b5b47d0a7405f735161210e (DEFAULT_TABLE table=test-table [id=15949659fc7a4af18af37f24aba96018]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:34:04.472663 8234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45ca50ef1b5b47d0a7405f735161210e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:34:04.472889 8103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45ca50ef1b5b47d0a7405f735161210e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:34:04.479771 8317 tablet_bootstrap.cc:492] T 45ca50ef1b5b47d0a7405f735161210e P 419671ebd8524b7aac4c3551a44a3e2a: Bootstrap starting.
I20260926 07:34:04.481178 8319 tablet_bootstrap.cc:492] T 45ca50ef1b5b47d0a7405f735161210e P e113b78131c94dae946ba53eb43d5f36: Bootstrap starting.
I20260926 07:34:04.482868 8317 tablet_bootstrap.cc:654] T 45ca50ef1b5b47d0a7405f735161210e P 419671ebd8524b7aac4c3551a44a3e2a: Neither blocks nor log segments found. Creating new log.
I20260926 07:34:04.483534 8319 tablet_bootstrap.cc:654] T 45ca50ef1b5b47d0a7405f735161210e P e113b78131c94dae946ba53eb43d5f36: Neither blocks nor log segments found. Creating new log.
I20260926 07:34:04.483914 8317 log.cc:824] T 45ca50ef1b5b47d0a7405f735161210e P 419671ebd8524b7aac4c3551a44a3e2a: Log is configured to *not* fsync() on all Append() calls
I20260926 07:34:04.484390 8319 log.cc:824] T 45ca50ef1b5b47d0a7405f735161210e P e113b78131c94dae946ba53eb43d5f36: Log is configured to *not* fsync() on all Append() calls
I20260926 07:34:04.484860 8318 tablet_bootstrap.cc:492] T 45ca50ef1b5b47d0a7405f735161210e P 3b9478c5d0c54ea2bcd246ce7d2db2c6: Bootstrap starting.
I20260926 07:34:04.486454 8319 tablet_bootstrap.cc:492] T 45ca50ef1b5b47d0a7405f735161210e P e113b78131c94dae946ba53eb43d5f36: No bootstrap required, opened a new log
I20260926 07:34:04.486573 8317 tablet_bootstrap.cc:492] T 45ca50ef1b5b47d0a7405f735161210e P 419671ebd8524b7aac4c3551a44a3e2a: No bootstrap required, opened a new log
I20260926 07:34:04.486727 8319 ts_tablet_manager.cc:1397] T 45ca50ef1b5b47d0a7405f735161210e P e113b78131c94dae946ba53eb43d5f36: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 07:34:04.486778 8317 ts_tablet_manager.cc:1397] T 45ca50ef1b5b47d0a7405f735161210e P 419671ebd8524b7aac4c3551a44a3e2a: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260926 07:34:04.487357 8318 tablet_bootstrap.cc:654] T 45ca50ef1b5b47d0a7405f735161210e P 3b9478c5d0c54ea2bcd246ce7d2db2c6: Neither blocks nor log segments found. Creating new log.
I20260926 07:34:04.488256 8318 log.cc:824] T 45ca50ef1b5b47d0a7405f735161210e P 3b9478c5d0c54ea2bcd246ce7d2db2c6: Log is configured to *not* fsync() on all Append() calls
I20260926 07:34:04.490787 8318 tablet_bootstrap.cc:492] T 45ca50ef1b5b47d0a7405f735161210e P 3b9478c5d0c54ea2bcd246ce7d2db2c6: No bootstrap required, opened a new log
I20260926 07:34:04.491019 8318 ts_tablet_manager.cc:1397] T 45ca50ef1b5b47d0a7405f735161210e P 3b9478c5d0c54ea2bcd246ce7d2db2c6: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 07:34:04.491165 8317 raft_consensus.cc:348] T 45ca50ef1b5b47d0a7405f735161210e P 419671ebd8524b7aac4c3551a44a3e2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b9478c5d0c54ea2bcd246ce7d2db2c6" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 38253 } } peers { permanent_uuid: "e113b78131c94dae946ba53eb43d5f36" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 39255 } } peers { permanent_uuid: "419671ebd8524b7aac4c3551a44a3e2a" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 40023 } }
I20260926 07:34:04.491072 8319 raft_consensus.cc:348] T 45ca50ef1b5b47d0a7405f735161210e P e113b78131c94dae946ba53eb43d5f36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b9478c5d0c54ea2bcd246ce7d2db2c6" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 38253 } } peers { permanent_uuid: "e113b78131c94dae946ba53eb43d5f36" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 39255 } } peers { permanent_uuid: "419671ebd8524b7aac4c3551a44a3e2a" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 40023 } }
I20260926 07:34:04.491534 8317 raft_consensus.cc:374] T 45ca50ef1b5b47d0a7405f735161210e P 419671ebd8524b7aac4c3551a44a3e2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:34:04.491560 8319 raft_consensus.cc:374] T 45ca50ef1b5b47d0a7405f735161210e P e113b78131c94dae946ba53eb43d5f36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:34:04.491601 8317 raft_consensus.cc:729] T 45ca50ef1b5b47d0a7405f735161210e P 419671ebd8524b7aac4c3551a44a3e2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 419671ebd8524b7aac4c3551a44a3e2a, State: Initialized, Role: FOLLOWER
I20260926 07:34:04.491617 8319 raft_consensus.cc:729] T 45ca50ef1b5b47d0a7405f735161210e P e113b78131c94dae946ba53eb43d5f36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e113b78131c94dae946ba53eb43d5f36, State: Initialized, Role: FOLLOWER
I20260926 07:34:04.492120 8319 consensus_queue.cc:260] T 45ca50ef1b5b47d0a7405f735161210e P e113b78131c94dae946ba53eb43d5f36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b9478c5d0c54ea2bcd246ce7d2db2c6" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 38253 } } peers { permanent_uuid: "e113b78131c94dae946ba53eb43d5f36" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 39255 } } peers { permanent_uuid: "419671ebd8524b7aac4c3551a44a3e2a" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 40023 } }
I20260926 07:34:04.492120 8317 consensus_queue.cc:260] T 45ca50ef1b5b47d0a7405f735161210e P 419671ebd8524b7aac4c3551a44a3e2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b9478c5d0c54ea2bcd246ce7d2db2c6" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 38253 } } peers { permanent_uuid: "e113b78131c94dae946ba53eb43d5f36" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 39255 } } peers { permanent_uuid: "419671ebd8524b7aac4c3551a44a3e2a" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 40023 } }
I20260926 07:34:04.493234 8319 ts_tablet_manager.cc:1428] T 45ca50ef1b5b47d0a7405f735161210e P e113b78131c94dae946ba53eb43d5f36: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260926 07:34:04.493266 8317 ts_tablet_manager.cc:1428] T 45ca50ef1b5b47d0a7405f735161210e P 419671ebd8524b7aac4c3551a44a3e2a: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260926 07:34:04.493651 8169 heartbeater.cc:503] Master 127.22.53.254:33571 was elected leader, sending a full tablet report...
I20260926 07:34:04.494014 8300 heartbeater.cc:503] Master 127.22.53.254:33571 was elected leader, sending a full tablet report...
I20260926 07:34:04.496722 8318 raft_consensus.cc:348] T 45ca50ef1b5b47d0a7405f735161210e P 3b9478c5d0c54ea2bcd246ce7d2db2c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b9478c5d0c54ea2bcd246ce7d2db2c6" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 38253 } } peers { permanent_uuid: "e113b78131c94dae946ba53eb43d5f36" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 39255 } } peers { permanent_uuid: "419671ebd8524b7aac4c3551a44a3e2a" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 40023 } }
I20260926 07:34:04.497082 8318 raft_consensus.cc:374] T 45ca50ef1b5b47d0a7405f735161210e P 3b9478c5d0c54ea2bcd246ce7d2db2c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:34:04.497184 8318 raft_consensus.cc:729] T 45ca50ef1b5b47d0a7405f735161210e P 3b9478c5d0c54ea2bcd246ce7d2db2c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b9478c5d0c54ea2bcd246ce7d2db2c6, State: Initialized, Role: FOLLOWER
I20260926 07:34:04.497723 8318 consensus_queue.cc:260] T 45ca50ef1b5b47d0a7405f735161210e P 3b9478c5d0c54ea2bcd246ce7d2db2c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b9478c5d0c54ea2bcd246ce7d2db2c6" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 38253 } } peers { permanent_uuid: "e113b78131c94dae946ba53eb43d5f36" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 39255 } } peers { permanent_uuid: "419671ebd8524b7aac4c3551a44a3e2a" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 40023 } }
I20260926 07:34:04.498533 8038 heartbeater.cc:503] Master 127.22.53.254:33571 was elected leader, sending a full tablet report...
I20260926 07:34:04.498967 8318 ts_tablet_manager.cc:1428] T 45ca50ef1b5b47d0a7405f735161210e P 3b9478c5d0c54ea2bcd246ce7d2db2c6: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
W20260926 07:34:04.525959 8039 tablet.cc:2369] T 45ca50ef1b5b47d0a7405f735161210e P 3b9478c5d0c54ea2bcd246ce7d2db2c6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:34:04.543613 8324 raft_consensus.cc:482] T 45ca50ef1b5b47d0a7405f735161210e P e113b78131c94dae946ba53eb43d5f36 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:34:04.543857 8324 raft_consensus.cc:504] T 45ca50ef1b5b47d0a7405f735161210e P e113b78131c94dae946ba53eb43d5f36 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b9478c5d0c54ea2bcd246ce7d2db2c6" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 38253 } } peers { permanent_uuid: "e113b78131c94dae946ba53eb43d5f36" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 39255 } } peers { permanent_uuid: "419671ebd8524b7aac4c3551a44a3e2a" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 40023 } }
I20260926 07:34:04.545163 8324 leader_election.cc:288] T 45ca50ef1b5b47d0a7405f735161210e P e113b78131c94dae946ba53eb43d5f36 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b9478c5d0c54ea2bcd246ce7d2db2c6 (127.22.53.193:38253), 419671ebd8524b7aac4c3551a44a3e2a (127.22.53.195:40023)
I20260926 07:34:04.548858 8171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:04.545462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.195:40023 (local address 127.22.53.194:49649)
0926 07:34:04.545593 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:04.545610 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:04.545710 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:04.546021 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:04.546024 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:04.546044 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:34:04.546172 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:04.546178 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:04.546905 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:04.546909 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:04.547707 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:04.547716 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:04.548369 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:04.548373 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:04.548474 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:04.548500 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 07:34:04.548633 (+ 133us) client_negotiation.cc:240] Negotiation successful
0926 07:34:04.548716 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260926 07:34:04.548858 8316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:04.545680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.194:49649 (local address 127.22.53.195:40023)
0926 07:34:04.545797 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:04.545800 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:04.545811 (+ 11us) server_negotiation.cc:407] Connection header received
0926 07:34:04.545862 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:04.545864 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:04.545903 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:04.546005 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:34:04.546288 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:04.546793 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:04.547844 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:04.548244 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:04.548370 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:04.548425 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:04.548652 (+ 227us) server_negotiation.cc:299] Negotiation successful
0926 07:34:04.548727 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 07:34:04.549093 8313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:04.545570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.194:42395 (local address 127.22.53.193:38253)
0926 07:34:04.545716 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:04.545722 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:04.546096 (+ 374us) server_negotiation.cc:407] Connection header received
0926 07:34:04.546242 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:04.546249 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:04.546293 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:04.546376 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:34:04.546630 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:04.547114 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:04.548127 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:04.548516 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:04.548684 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:04.548732 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:04.548907 (+ 175us) server_negotiation.cc:299] Negotiation successful
0926 07:34:04.548986 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 07:34:04.549105 8326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:04.545462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.193:38253 (local address 127.22.53.194:42395)
0926 07:34:04.545996 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:04.546008 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:04.546121 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:04.546386 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:04.546389 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:04.546404 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:34:04.546520 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:04.546526 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:04.547225 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:04.547231 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:04.547998 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:04.548006 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:04.548679 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:04.548682 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:04.548781 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:04.548806 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 07:34:04.548891 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 07:34:04.548957 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"mutex_wait_us":138,"thread_start_us":184,"threads_started":1}
I20260926 07:34:04.549643 8254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45ca50ef1b5b47d0a7405f735161210e" candidate_uuid: "e113b78131c94dae946ba53eb43d5f36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "419671ebd8524b7aac4c3551a44a3e2a" is_pre_election: true
I20260926 07:34:04.549772 7992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45ca50ef1b5b47d0a7405f735161210e" candidate_uuid: "e113b78131c94dae946ba53eb43d5f36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b9478c5d0c54ea2bcd246ce7d2db2c6" is_pre_election: true
I20260926 07:34:04.550007 8254 raft_consensus.cc:2454] T 45ca50ef1b5b47d0a7405f735161210e P 419671ebd8524b7aac4c3551a44a3e2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e113b78131c94dae946ba53eb43d5f36 in term 0.
I20260926 07:34:04.550087 7992 raft_consensus.cc:2454] T 45ca50ef1b5b47d0a7405f735161210e P 3b9478c5d0c54ea2bcd246ce7d2db2c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e113b78131c94dae946ba53eb43d5f36 in term 0.
I20260926 07:34:04.550622 8056 leader_election.cc:302] T 45ca50ef1b5b47d0a7405f735161210e P e113b78131c94dae946ba53eb43d5f36 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3b9478c5d0c54ea2bcd246ce7d2db2c6, 419671ebd8524b7aac4c3551a44a3e2a, e113b78131c94dae946ba53eb43d5f36; no voters:
I20260926 07:34:04.550938 8324 raft_consensus.cc:2781] T 45ca50ef1b5b47d0a7405f735161210e P e113b78131c94dae946ba53eb43d5f36 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:34:04.551025 8324 raft_consensus.cc:482] T 45ca50ef1b5b47d0a7405f735161210e P e113b78131c94dae946ba53eb43d5f36 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:34:04.551074 8324 raft_consensus.cc:3037] T 45ca50ef1b5b47d0a7405f735161210e P e113b78131c94dae946ba53eb43d5f36 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:34:04.552156 8324 raft_consensus.cc:504] T 45ca50ef1b5b47d0a7405f735161210e P e113b78131c94dae946ba53eb43d5f36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b9478c5d0c54ea2bcd246ce7d2db2c6" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 38253 } } peers { permanent_uuid: "e113b78131c94dae946ba53eb43d5f36" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 39255 } } peers { permanent_uuid: "419671ebd8524b7aac4c3551a44a3e2a" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 40023 } }
I20260926 07:34:04.552534 8324 leader_election.cc:288] T 45ca50ef1b5b47d0a7405f735161210e P e113b78131c94dae946ba53eb43d5f36 [CANDIDATE]: Term 1 election: Requested vote from peers 3b9478c5d0c54ea2bcd246ce7d2db2c6 (127.22.53.193:38253), 419671ebd8524b7aac4c3551a44a3e2a (127.22.53.195:40023)
I20260926 07:34:04.552985 8254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45ca50ef1b5b47d0a7405f735161210e" candidate_uuid: "e113b78131c94dae946ba53eb43d5f36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "419671ebd8524b7aac4c3551a44a3e2a"
I20260926 07:34:04.552986 7992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45ca50ef1b5b47d0a7405f735161210e" candidate_uuid: "e113b78131c94dae946ba53eb43d5f36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b9478c5d0c54ea2bcd246ce7d2db2c6"
I20260926 07:34:04.553135 8254 raft_consensus.cc:3037] T 45ca50ef1b5b47d0a7405f735161210e P 419671ebd8524b7aac4c3551a44a3e2a [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:34:04.553136 7992 raft_consensus.cc:3037] T 45ca50ef1b5b47d0a7405f735161210e P 3b9478c5d0c54ea2bcd246ce7d2db2c6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:34:04.554239 8254 raft_consensus.cc:2454] T 45ca50ef1b5b47d0a7405f735161210e P 419671ebd8524b7aac4c3551a44a3e2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e113b78131c94dae946ba53eb43d5f36 in term 1.
I20260926 07:34:04.554351 7992 raft_consensus.cc:2454] T 45ca50ef1b5b47d0a7405f735161210e P 3b9478c5d0c54ea2bcd246ce7d2db2c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e113b78131c94dae946ba53eb43d5f36 in term 1.
I20260926 07:34:04.554634 8056 leader_election.cc:302] T 45ca50ef1b5b47d0a7405f735161210e P e113b78131c94dae946ba53eb43d5f36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 419671ebd8524b7aac4c3551a44a3e2a, e113b78131c94dae946ba53eb43d5f36; no voters:
I20260926 07:34:04.554831 8324 raft_consensus.cc:2781] T 45ca50ef1b5b47d0a7405f735161210e P e113b78131c94dae946ba53eb43d5f36 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:34:04.555068 8324 raft_consensus.cc:686] T 45ca50ef1b5b47d0a7405f735161210e P e113b78131c94dae946ba53eb43d5f36 [term 1 LEADER]: Becoming Leader. State: Replica: e113b78131c94dae946ba53eb43d5f36, State: Running, Role: LEADER
I20260926 07:34:04.555435 8324 consensus_queue.cc:237] T 45ca50ef1b5b47d0a7405f735161210e P e113b78131c94dae946ba53eb43d5f36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b9478c5d0c54ea2bcd246ce7d2db2c6" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 38253 } } peers { permanent_uuid: "e113b78131c94dae946ba53eb43d5f36" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 39255 } } peers { permanent_uuid: "419671ebd8524b7aac4c3551a44a3e2a" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 40023 } }
I20260926 07:34:04.559155 7850 catalog_manager.cc:5495] T 45ca50ef1b5b47d0a7405f735161210e P e113b78131c94dae946ba53eb43d5f36 reported cstate change: term changed from 0 to 1, leader changed from <none> to e113b78131c94dae946ba53eb43d5f36 (127.22.53.194). New cstate: current_term: 1 leader_uuid: "e113b78131c94dae946ba53eb43d5f36" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b9478c5d0c54ea2bcd246ce7d2db2c6" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 38253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e113b78131c94dae946ba53eb43d5f36" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 39255 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "419671ebd8524b7aac4c3551a44a3e2a" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 40023 } health_report { overall_health: UNKNOWN } } }
I20260926 07:34:04.594676 8314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:04.590896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41506 (local address 127.22.53.194:39255)
0926 07:34:04.591039 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:04.591044 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:04.591082 (+ 38us) server_negotiation.cc:407] Connection header received
0926 07:34:04.591187 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:04.591190 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:04.591265 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:04.591378 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:34:04.591613 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:04.592185 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:04.593100 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:04.593333 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:04.593440 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:04.593773 (+ 333us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:34:04.594231 (+ 458us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:34:04.594363 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:04.594427 (+ 64us) server_negotiation.cc:299] Negotiation successful
0926 07:34:04.594484 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 07:34:04.601123 8254 raft_consensus.cc:1261] T 45ca50ef1b5b47d0a7405f735161210e P 419671ebd8524b7aac4c3551a44a3e2a [term 1 FOLLOWER]: Refusing update from remote peer e113b78131c94dae946ba53eb43d5f36: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:34:04.601123 7992 raft_consensus.cc:1261] T 45ca50ef1b5b47d0a7405f735161210e P 3b9478c5d0c54ea2bcd246ce7d2db2c6 [term 1 FOLLOWER]: Refusing update from remote peer e113b78131c94dae946ba53eb43d5f36: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:34:04.601948 8327 consensus_queue.cc:1035] T 45ca50ef1b5b47d0a7405f735161210e P e113b78131c94dae946ba53eb43d5f36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "419671ebd8524b7aac4c3551a44a3e2a" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 40023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:34:04.602147 8324 consensus_queue.cc:1035] T 45ca50ef1b5b47d0a7405f735161210e P e113b78131c94dae946ba53eb43d5f36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9478c5d0c54ea2bcd246ce7d2db2c6" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 38253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:34:04.609226 8330 mvcc.cc:205] Tried to move back new op lower bound from 7333511350678102016 to 7333511350501883904. Current Snapshot: MvccSnapshot[applied={T|T < 7333511350678102016}]
I20260926 07:34:04.610760 8332 mvcc.cc:205] Tried to move back new op lower bound from 7333511350678102016 to 7333511350501883904. Current Snapshot: MvccSnapshot[applied={T|T < 7333511350678102016}]
I20260926 07:34:04.610810 8331 mvcc.cc:205] Tried to move back new op lower bound from 7333511350678102016 to 7333511350501883904. Current Snapshot: MvccSnapshot[applied={T|T < 7333511350678102016}]
I20260926 07:34:04.619266 7849 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39366:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:34:04.619474 7849 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39366:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:34:04.622277 7849 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 74b215d152f34aac954fc55e3f2850fb: Sending DeleteTablet for 3 replicas of tablet 45ca50ef1b5b47d0a7405f735161210e
I20260926 07:34:04.623205 7972 tablet_service.cc:1514] Processing DeleteTablet for tablet 45ca50ef1b5b47d0a7405f735161210e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:34:04 UTC) from {username='slave'} at 127.0.0.1:53330
I20260926 07:34:04.623201 8234 tablet_service.cc:1514] Processing DeleteTablet for tablet 45ca50ef1b5b47d0a7405f735161210e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:34:04 UTC) from {username='slave'} at 127.0.0.1:37124
I20260926 07:34:04.623273 8103 tablet_service.cc:1514] Processing DeleteTablet for tablet 45ca50ef1b5b47d0a7405f735161210e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:34:04 UTC) from {username='slave'} at 127.0.0.1:41494
I20260926 07:34:04.623859 8340 tablet_replica.cc:330] T 45ca50ef1b5b47d0a7405f735161210e P 419671ebd8524b7aac4c3551a44a3e2a: stopping tablet replica
I20260926 07:34:04.624095 8341 tablet_replica.cc:330] T 45ca50ef1b5b47d0a7405f735161210e P 3b9478c5d0c54ea2bcd246ce7d2db2c6: stopping tablet replica
I20260926 07:34:04.624311 8340 raft_consensus.cc:2229] T 45ca50ef1b5b47d0a7405f735161210e P 419671ebd8524b7aac4c3551a44a3e2a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:34:04.624421 8341 raft_consensus.cc:2229] T 45ca50ef1b5b47d0a7405f735161210e P 3b9478c5d0c54ea2bcd246ce7d2db2c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:34:04.624750 8340 raft_consensus.cc:2258] T 45ca50ef1b5b47d0a7405f735161210e P 419671ebd8524b7aac4c3551a44a3e2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:34:04.624765 8341 raft_consensus.cc:2258] T 45ca50ef1b5b47d0a7405f735161210e P 3b9478c5d0c54ea2bcd246ce7d2db2c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:34:04.624907 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 7910
I20260926 07:34:04.625593 8342 tablet_replica.cc:330] T 45ca50ef1b5b47d0a7405f735161210e P e113b78131c94dae946ba53eb43d5f36: stopping tablet replica
I20260926 07:34:04.625923 8342 raft_consensus.cc:2229] T 45ca50ef1b5b47d0a7405f735161210e P e113b78131c94dae946ba53eb43d5f36 [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:34:04.626374 8342 raft_consensus.cc:2258] T 45ca50ef1b5b47d0a7405f735161210e P e113b78131c94dae946ba53eb43d5f36 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:34:04.626889 8340 ts_tablet_manager.cc:1905] T 45ca50ef1b5b47d0a7405f735161210e P 419671ebd8524b7aac4c3551a44a3e2a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:34:04.628391 8342 ts_tablet_manager.cc:1905] T 45ca50ef1b5b47d0a7405f735161210e P e113b78131c94dae946ba53eb43d5f36: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:34:04.630688 8340 ts_tablet_manager.cc:1918] T 45ca50ef1b5b47d0a7405f735161210e P 419671ebd8524b7aac4c3551a44a3e2a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:34:04.630908 8340 log.cc:1196] T 45ca50ef1b5b47d0a7405f735161210e P 419671ebd8524b7aac4c3551a44a3e2a: Deleting WAL directory at /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407783529717-22743-0/minicluster-data/ts-2/wal/wals/45ca50ef1b5b47d0a7405f735161210e
I20260926 07:34:04.631142 8342 ts_tablet_manager.cc:1918] T 45ca50ef1b5b47d0a7405f735161210e P e113b78131c94dae946ba53eb43d5f36: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:34:04.631247 8342 log.cc:1196] T 45ca50ef1b5b47d0a7405f735161210e P e113b78131c94dae946ba53eb43d5f36: Deleting WAL directory at /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407783529717-22743-0/minicluster-data/ts-1/wal/wals/45ca50ef1b5b47d0a7405f735161210e
I20260926 07:34:04.631448 8340 ts_tablet_manager.cc:1939] T 45ca50ef1b5b47d0a7405f735161210e P 419671ebd8524b7aac4c3551a44a3e2a: Deleting consensus metadata
I20260926 07:34:04.631557 8342 ts_tablet_manager.cc:1939] T 45ca50ef1b5b47d0a7405f735161210e P e113b78131c94dae946ba53eb43d5f36: Deleting consensus metadata
I20260926 07:34:04.633016 7834 catalog_manager.cc:4841] TS 419671ebd8524b7aac4c3551a44a3e2a (127.22.53.195:40023): tablet 45ca50ef1b5b47d0a7405f735161210e (table test-table [id=15949659fc7a4af18af37f24aba96018]) successfully deleted
I20260926 07:34:04.633102 7837 catalog_manager.cc:4841] TS e113b78131c94dae946ba53eb43d5f36 (127.22.53.194:39255): tablet 45ca50ef1b5b47d0a7405f735161210e (table test-table [id=15949659fc7a4af18af37f24aba96018]) successfully deleted
W20260926 07:34:04.633457 7836 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.53.193:38253 (error 108)
I20260926 07:34:04.633713 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 8041
I20260926 07:34:04.635751 8311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:04.635336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.193:38253 (local address 127.0.0.1:53334)
0926 07:34:04.635525 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:04.635629 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.193:38253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260926 07:34:04.635960 7836 catalog_manager.cc:4584] TS 3b9478c5d0c54ea2bcd246ce7d2db2c6 (127.22.53.193:38253): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 45ca50ef1b5b47d0a7405f735161210e: Network error: Client connection negotiation failed: client connection to 127.22.53.193:38253: connect: Connection refused (error 111)
I20260926 07:34:04.639792 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 8172
I20260926 07:34:04.645609 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 7819
2026-09-26T07:34:04Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1143 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407783529717-22743-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:34:04 dist-test-slave-wdjc krb5kdc[8346](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:34:04 dist-test-slave-wdjc krb5kdc[8346](info): set up 2 sockets
Sep 26 07:34:04 dist-test-slave-wdjc krb5kdc[8346](info): commencing operation
krb5kdc: starting...
W20260926 07:34:05.708694 22743 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407783529717-22743-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407783529717-22743-0/krb5kdc/test-user.keytab.
Sep 26 07:34:05 dist-test-slave-wdjc krb5kdc[8346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408045, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:34:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:34:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.53.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.53.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.53.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab.
Entry for principal kudu/127.22.53.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab.
I20260926 07:34:05.841872 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407783529717-22743-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407783529717-22743-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407783529717-22743-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.53.254:38243
--webserver_interface=127.22.53.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab
--principal=kudu/127.22.53.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:44991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.53.254:38243
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:34:05.963402 8359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:34:05.963790 8359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:34:05.963871 8359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:34:05.968180 8359 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:34:05.968307 8359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:34:05.968345 8359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:34:05.968387 8359 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:34:05.968422 8359 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:34:05.974189 8359 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:44991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407783529717-22743-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407783529717-22743-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.53.254:38243
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab
--principal=kudu/127.22.53.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.53.254:38243
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.53.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8359
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407783529717-22743-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:34:05.975430 8359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:34:05.976589 8359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:34:05.985782 8367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:05.985916 8364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:34:05.985947 8359 server_base.cc:1034] running on GCE node
W20260926 07:34:05.986140 8365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:34:05.986809 8359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:34:05.988025 8359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:34:05.989230 8359 hybrid_clock.cc:648] HybridClock initialized: now 1790408045989178 us; error 66 us; skew 500 ppm
Sep 26 07:34:05 dist-test-slave-wdjc krb5kdc[8346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408045, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:34:05.992362 8359 init.cc:377] Logged in from keytab as kudu/127.22.53.254@KRBTEST.COM (short username kudu)
I20260926 07:34:05.993503 8359 webserver.cc:460] Webserver started at http://127.22.53.254:43039/ using document root <none> and password file <none>
I20260926 07:34:05.994139 8359 fs_manager.cc:359] Metadata directory not provided
I20260926 07:34:05.994241 8359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:34:05.994510 8359 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:34:05.996873 8359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407783529717-22743-0/minicluster-data/master-0/data/instance:
uuid: "0d614df8bdf94a518700fd14f0687e77"
format_stamp: "Formatted at 2026-09-26 07:34:05 on dist-test-slave-wdjc"
server_key: "84b5d272112963e5b416d14e6595d9a6"
server_key_iv: "81ce45eb8fe59f6c92538a1f39330087"
server_key_version: "encryptionkey@0"
I20260926 07:34:05.997509 8359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407783529717-22743-0/minicluster-data/master-0/wal/instance:
uuid: "0d614df8bdf94a518700fd14f0687e77"
format_stamp: "Formatted at 2026-09-26 07:34:05 on dist-test-slave-wdjc"
server_key: "84b5d272112963e5b416d14e6595d9a6"
server_key_iv: "81ce45eb8fe59f6c92538a1f39330087"
server_key_version: "encryptionkey@0"
I20260926 07:34:06.001979 8359 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 07:34:06.004853 8374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:06.006511 8359 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:34:06.006709 8359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407783529717-22743-0/minicluster-data/master-0/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407783529717-22743-0/minicluster-data/master-0/wal
uuid: "0d614df8bdf94a518700fd14f0687e77"
format_stamp: "Formatted at 2026-09-26 07:34:05 on dist-test-slave-wdjc"
server_key: "84b5d272112963e5b416d14e6595d9a6"
server_key_iv: "81ce45eb8fe59f6c92538a1f39330087"
server_key_version: "encryptionkey@0"
I20260926 07:34:06.006866 8359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407783529717-22743-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407783529717-22743-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407783529717-22743-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:34:06.044581 8359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:34:06.048262 8359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:34:06.048513 8359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:34:06.056273 8359 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.254:38243
I20260926 07:34:06.056299 8426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.254:38243 every 8 connection(s)
I20260926 07:34:06.057442 8359 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
I20260926 07:34:06.059566 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 8359
I20260926 07:34:06.059725 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407783529717-22743-0/minicluster-data/master-0/wal/instance
I20260926 07:34:06.060024 22743 external_mini_cluster.cc:1428] Setting key ae9ff8583b0349cf9e3cfb644fbff38c
I20260926 07:34:06.060575 8427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:34:06.068354 8427 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d614df8bdf94a518700fd14f0687e77: Bootstrap starting.
Sep 26 07:34:06 dist-test-slave-wdjc krb5kdc[8346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408045, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.53.254@KRBTEST.COM
I20260926 07:34:06.071311 8427 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d614df8bdf94a518700fd14f0687e77: Neither blocks nor log segments found. Creating new log.
I20260926 07:34:06.072365 8427 log.cc:824] T 00000000000000000000000000000000 P 0d614df8bdf94a518700fd14f0687e77: Log is configured to *not* fsync() on all Append() calls
I20260926 07:34:06.075059 8427 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d614df8bdf94a518700fd14f0687e77: No bootstrap required, opened a new log
I20260926 07:34:06.077489 8430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:06.061570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51560 (local address 127.22.53.254:38243)
0926 07:34:06.062080 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:06.062100 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:06.062143 (+ 43us) server_negotiation.cc:407] Connection header received
0926 07:34:06.062979 (+ 836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:06.063009 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:06.063469 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:06.063886 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:06.064847 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:06.066357 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:06.067496 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:06.068214 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:06.068393 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:06.071459 (+ 3066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:06.071465 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:06.071487 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:06.071532 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:06.073689 (+ 2157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:06.074280 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:06.074286 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:06.074293 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:06.074441 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:06.074738 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:06.074741 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:06.074742 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:06.075300 (+ 558us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:06.075518 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:06.075833 (+ 315us) server_negotiation.cc:299] Negotiation successful
0926 07:34:06.076187 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":152,"threads_started":1}
I20260926 07:34:06.079290 8427 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d614df8bdf94a518700fd14f0687e77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d614df8bdf94a518700fd14f0687e77" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 38243 } }
I20260926 07:34:06.079531 8427 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d614df8bdf94a518700fd14f0687e77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:34:06.079566 8427 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d614df8bdf94a518700fd14f0687e77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d614df8bdf94a518700fd14f0687e77, State: Initialized, Role: FOLLOWER
I20260926 07:34:06.080013 8427 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d614df8bdf94a518700fd14f0687e77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d614df8bdf94a518700fd14f0687e77" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 38243 } }
I20260926 07:34:06.080138 8427 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d614df8bdf94a518700fd14f0687e77 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:34:06.080181 8427 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d614df8bdf94a518700fd14f0687e77 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:34:06.080312 8427 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d614df8bdf94a518700fd14f0687e77 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:34:06.081403 8427 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d614df8bdf94a518700fd14f0687e77 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d614df8bdf94a518700fd14f0687e77" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 38243 } }
I20260926 07:34:06.081779 8427 leader_election.cc:302] T 00000000000000000000000000000000 P 0d614df8bdf94a518700fd14f0687e77 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d614df8bdf94a518700fd14f0687e77; no voters:
I20260926 07:34:06.082157 8427 leader_election.cc:288] T 00000000000000000000000000000000 P 0d614df8bdf94a518700fd14f0687e77 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:34:06.082294 8432 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d614df8bdf94a518700fd14f0687e77 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:34:06.082548 8432 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d614df8bdf94a518700fd14f0687e77 [term 1 LEADER]: Becoming Leader. State: Replica: 0d614df8bdf94a518700fd14f0687e77, State: Running, Role: LEADER
I20260926 07:34:06.082870 8432 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d614df8bdf94a518700fd14f0687e77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d614df8bdf94a518700fd14f0687e77" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 38243 } }
I20260926 07:34:06.083479 8427 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d614df8bdf94a518700fd14f0687e77 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:34:06.084503 8434 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d614df8bdf94a518700fd14f0687e77 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d614df8bdf94a518700fd14f0687e77. Latest consensus state: current_term: 1 leader_uuid: "0d614df8bdf94a518700fd14f0687e77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d614df8bdf94a518700fd14f0687e77" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 38243 } } }
I20260926 07:34:06.084676 8434 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d614df8bdf94a518700fd14f0687e77 [sys.catalog]: This master's current role is: LEADER
I20260926 07:34:06.084848 8433 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d614df8bdf94a518700fd14f0687e77 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0d614df8bdf94a518700fd14f0687e77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d614df8bdf94a518700fd14f0687e77" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 38243 } } }
I20260926 07:34:06.084947 8433 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d614df8bdf94a518700fd14f0687e77 [sys.catalog]: This master's current role is: LEADER
I20260926 07:34:06.086099 8441 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:34:06.089243 8441 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:34:06.095532 8441 catalog_manager.cc:1347] Generated new cluster ID: 0cf04ca043124153b636fe61d04c04ea
I20260926 07:34:06.095643 8441 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:34:06.104619 8441 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:34:06.105981 8441 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:34:06.118431 8441 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d614df8bdf94a518700fd14f0687e77: Generated new TSK 0
I20260926 07:34:06.119390 8441 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.53.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.53.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.53.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.193.keytab.
Entry for principal kudu/127.22.53.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.193.keytab.
I20260926 07:34:06.161980 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407783529717-22743-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407783529717-22743-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407783529717-22743-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407783529717-22743-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.53.193:0
--local_ip_for_outbound_sockets=127.22.53.193
--webserver_interface=127.22.53.193
--webserver_port=0
--tserver_master_addrs=127.22.53.254:38243
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.193.keytab
--principal=kudu/127.22.53.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:44991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:34:06.294409 8453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:34:06.294725 8453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:34:06.294773 8453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:34:06.299055 8453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:34:06.299131 8453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:34:06.299196 8453 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.53.193
I20260926 07:34:06.304934 8453 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:44991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407783529717-22743-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407783529717-22743-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.193.keytab
--principal=kudu/127.22.53.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.53.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407783529717-22743-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.53.193
--webserver_port=0
--tserver_master_addrs=127.22.53.254:38243
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8453
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.53.193
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407783529717-22743-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:34:06.306063 8453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:34:06.307082 8453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:34:06.317737 8461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:06.317737 8459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:06.317929 8458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:34:06.318200 8453 server_base.cc:1034] running on GCE node
I20260926 07:34:06.318681 8453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:34:06.319480 8453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:34:06.320688 8453 hybrid_clock.cc:648] HybridClock initialized: now 1790408046320652 us; error 57 us; skew 500 ppm
Sep 26 07:34:06 dist-test-slave-wdjc krb5kdc[8346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408046, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:34:06.323872 8453 init.cc:377] Logged in from keytab as kudu/127.22.53.193@KRBTEST.COM (short username kudu)
I20260926 07:34:06.324915 8453 webserver.cc:460] Webserver started at http://127.22.53.193:34025/ using document root <none> and password file <none>
I20260926 07:34:06.325500 8453 fs_manager.cc:359] Metadata directory not provided
I20260926 07:34:06.325562 8453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:34:06.325815 8453 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:34:06.328051 8453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407783529717-22743-0/minicluster-data/ts-0/data/instance:
uuid: "aa3a909d1b614776a0a8f7b27cd0b35c"
format_stamp: "Formatted at 2026-09-26 07:34:06 on dist-test-slave-wdjc"
server_key: "ff15f7a7887f3a87efdcff9a520a8c9f"
server_key_iv: "dbeef4fa805e335962f62d10ca761dce"
server_key_version: "encryptionkey@0"
I20260926 07:34:06.328688 8453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407783529717-22743-0/minicluster-data/ts-0/wal/instance:
uuid: "aa3a909d1b614776a0a8f7b27cd0b35c"
format_stamp: "Formatted at 2026-09-26 07:34:06 on dist-test-slave-wdjc"
server_key: "ff15f7a7887f3a87efdcff9a520a8c9f"
server_key_iv: "dbeef4fa805e335962f62d10ca761dce"
server_key_version: "encryptionkey@0"
I20260926 07:34:06.333009 8453 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 07:34:06.335958 8468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:06.337483 8453 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:34:06.337618 8453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407783529717-22743-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407783529717-22743-0/minicluster-data/ts-0/wal
uuid: "aa3a909d1b614776a0a8f7b27cd0b35c"
format_stamp: "Formatted at 2026-09-26 07:34:06 on dist-test-slave-wdjc"
server_key: "ff15f7a7887f3a87efdcff9a520a8c9f"
server_key_iv: "dbeef4fa805e335962f62d10ca761dce"
server_key_version: "encryptionkey@0"
I20260926 07:34:06.337733 8453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407783529717-22743-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407783529717-22743-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407783529717-22743-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:34:06.355996 8453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:34:06.359833 8453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:34:06.360067 8453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:34:06.360752 8453 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:34:06.361670 8453 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:34:06.361756 8453 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:06.361830 8453 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:34:06.361892 8453 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:06.372144 8453 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.193:41295
I20260926 07:34:06.372203 8581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.193:41295 every 8 connection(s)
I20260926 07:34:06.373207 8453 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407783529717-22743-0/minicluster-data/ts-0/data/info.pb
I20260926 07:34:06.380671 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 8453
I20260926 07:34:06.380827 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407783529717-22743-0/minicluster-data/ts-0/wal/instance
I20260926 07:34:06.381127 22743 external_mini_cluster.cc:1428] Setting key d53fdd8da25510adc5f6d5b07820a6b5
Sep 26 07:34:06 dist-test-slave-wdjc krb5kdc[8346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408046, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.193@KRBTEST.COM for kudu/127.22.53.254@KRBTEST.COM
I20260926 07:34:06.389775 8430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:06.375312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.193:42103 (local address 127.22.53.254:38243)
0926 07:34:06.375505 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:06.375510 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:06.376401 (+ 891us) server_negotiation.cc:407] Connection header received
0926 07:34:06.377931 (+ 1530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:06.377934 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:06.377986 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:06.378075 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:06.379340 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:06.379909 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:06.381535 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:06.381787 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:06.381941 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:06.385163 (+ 3222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:06.385168 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:06.385171 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:06.385209 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:06.387073 (+ 1864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:06.387764 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:06.387770 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:06.387775 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:06.387885 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:06.388280 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:06.388286 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:06.388291 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:06.388624 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:06.388749 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:06.389371 (+ 622us) server_negotiation.cc:299] Negotiation successful
0926 07:34:06.389550 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260926 07:34:06.390610 8584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:06.375681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:38243 (local address 127.22.53.193:42103)
0926 07:34:06.376211 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:06.376273 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:06.377699 (+ 1426us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:06.378277 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:06.378285 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:06.378664 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:06.379171 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:06.379185 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:06.380072 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:06.380076 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:06.381390 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:06.381403 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:06.381990 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:06.381994 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:06.382200 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:06.382849 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:06.382866 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:06.384940 (+ 2074us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:34:06.387220 (+ 2280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:06.387226 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:06.387240 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:06.387607 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:06.387996 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:06.387999 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:06.388000 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:06.388160 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:06.388770 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:34:06.388776 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:34:06.389071 (+ 295us) client_negotiation.cc:769] Sending connection context
0926 07:34:06.389265 (+ 194us) client_negotiation.cc:240] Negotiation successful
0926 07:34:06.389573 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":121,"threads_started":1}
WARNING: no policy specified for kudu/127.22.53.194@KRBTEST.COM; defaulting to no policy
I20260926 07:34:06.392061 8582 heartbeater.cc:347] Connected to a master server at 127.22.53.254:38243
I20260926 07:34:06.392338 8582 heartbeater.cc:464] Registering TS with master...
I20260926 07:34:06.393002 8582 heartbeater.cc:511] Master 127.22.53.254:38243 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.53.194@KRBTEST.COM" created.
I20260926 07:34:06.394727 8391 ts_manager.cc:194] Registered new tserver with Master: aa3a909d1b614776a0a8f7b27cd0b35c (127.22.53.193:41295)
I20260926 07:34:06.396445 8391 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.53.193@KRBTEST.COM'} at 127.22.53.193:42103
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.53.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.194.keytab.
Entry for principal kudu/127.22.53.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.194.keytab.
I20260926 07:34:06.412256 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407783529717-22743-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407783529717-22743-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407783529717-22743-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407783529717-22743-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.53.194:0
--local_ip_for_outbound_sockets=127.22.53.194
--webserver_interface=127.22.53.194
--webserver_port=0
--tserver_master_addrs=127.22.53.254:38243
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.194.keytab
--principal=kudu/127.22.53.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:44991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:34:06.528286 8587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:34:06.528617 8587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:34:06.528702 8587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:34:06.532852 8587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:34:06.532925 8587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:34:06.533025 8587 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.53.194
I20260926 07:34:06.538527 8587 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:44991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407783529717-22743-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407783529717-22743-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.194.keytab
--principal=kudu/127.22.53.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.53.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407783529717-22743-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.53.194
--webserver_port=0
--tserver_master_addrs=127.22.53.254:38243
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8587
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.53.194
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407783529717-22743-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:34:06.539709 8587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:34:06.540817 8587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:34:06.551113 8593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:06.551117 8592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:06.551448 8595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:34:06.551448 8587 server_base.cc:1034] running on GCE node
I20260926 07:34:06.551924 8587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:34:06.552696 8587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:34:06.553881 8587 hybrid_clock.cc:648] HybridClock initialized: now 1790408046553858 us; error 37 us; skew 500 ppm
Sep 26 07:34:06 dist-test-slave-wdjc krb5kdc[8346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408046, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:34:06.557060 8587 init.cc:377] Logged in from keytab as kudu/127.22.53.194@KRBTEST.COM (short username kudu)
I20260926 07:34:06.558039 8587 webserver.cc:460] Webserver started at http://127.22.53.194:42345/ using document root <none> and password file <none>
I20260926 07:34:06.558647 8587 fs_manager.cc:359] Metadata directory not provided
I20260926 07:34:06.558715 8587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:34:06.558940 8587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:34:06.561208 8587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407783529717-22743-0/minicluster-data/ts-1/data/instance:
uuid: "09f9b7b30d5c46ea94ec98aa03d1f6bb"
format_stamp: "Formatted at 2026-09-26 07:34:06 on dist-test-slave-wdjc"
server_key: "5b2c6b28fcf273db1f13007054b9b3c6"
server_key_iv: "acdbfee9a73e820a91b24f416b097f30"
server_key_version: "encryptionkey@0"
I20260926 07:34:06.561806 8587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407783529717-22743-0/minicluster-data/ts-1/wal/instance:
uuid: "09f9b7b30d5c46ea94ec98aa03d1f6bb"
format_stamp: "Formatted at 2026-09-26 07:34:06 on dist-test-slave-wdjc"
server_key: "5b2c6b28fcf273db1f13007054b9b3c6"
server_key_iv: "acdbfee9a73e820a91b24f416b097f30"
server_key_version: "encryptionkey@0"
I20260926 07:34:06.566057 8587 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 07:34:06.568969 8602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:06.570374 8587 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 07:34:06.570552 8587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407783529717-22743-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407783529717-22743-0/minicluster-data/ts-1/wal
uuid: "09f9b7b30d5c46ea94ec98aa03d1f6bb"
format_stamp: "Formatted at 2026-09-26 07:34:06 on dist-test-slave-wdjc"
server_key: "5b2c6b28fcf273db1f13007054b9b3c6"
server_key_iv: "acdbfee9a73e820a91b24f416b097f30"
server_key_version: "encryptionkey@0"
I20260926 07:34:06.570700 8587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407783529717-22743-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407783529717-22743-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407783529717-22743-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:34:06.595954 8587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:34:06.599846 8587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:34:06.600214 8587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:34:06.600991 8587 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:34:06.601907 8587 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:34:06.601985 8587 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:06.602061 8587 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:34:06.602106 8587 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:06.611136 8587 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.194:41173
I20260926 07:34:06.611163 8715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.194:41173 every 8 connection(s)
I20260926 07:34:06.612262 8587 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407783529717-22743-0/minicluster-data/ts-1/data/info.pb
I20260926 07:34:06.619768 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 8587
I20260926 07:34:06.619906 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407783529717-22743-0/minicluster-data/ts-1/wal/instance
I20260926 07:34:06.620209 22743 external_mini_cluster.cc:1428] Setting key 71064102d6d859f135392a5a7e9399ec
Sep 26 07:34:06 dist-test-slave-wdjc krb5kdc[8346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408046, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.194@KRBTEST.COM for kudu/127.22.53.254@KRBTEST.COM
I20260926 07:34:06.628822 8430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:06.614387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.194:48183 (local address 127.22.53.254:38243)
0926 07:34:06.614534 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:06.614538 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:06.615474 (+ 936us) server_negotiation.cc:407] Connection header received
0926 07:34:06.616954 (+ 1480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:06.616957 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:06.617024 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:06.617145 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:06.618452 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:06.619028 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:06.620495 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:06.620885 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:06.620998 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:06.624108 (+ 3110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:06.624115 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:06.624121 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:06.624156 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:06.625845 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:06.626618 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:06.626623 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:06.626625 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:06.626726 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:06.627209 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:06.627212 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:06.627213 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:06.627440 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:06.627539 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:06.628489 (+ 950us) server_negotiation.cc:299] Negotiation successful
0926 07:34:06.628686 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 07:34:06.630125 8718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:06.614773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:38243 (local address 127.22.53.194:48183)
0926 07:34:06.615290 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:06.615341 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:06.616719 (+ 1378us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:06.617327 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:06.617334 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:06.617720 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:06.618277 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:06.618295 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:06.619187 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:06.619193 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:06.620306 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:06.620321 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:06.621064 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:06.621068 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:06.621270 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:06.622008 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:06.622032 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:06.623890 (+ 1858us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:34:06.625991 (+ 2101us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:06.626001 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:06.626023 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:06.626484 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:06.626842 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:06.626848 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:06.626852 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:06.627075 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:06.627569 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:34:06.627581 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:34:06.628120 (+ 539us) client_negotiation.cc:769] Sending connection context
0926 07:34:06.628407 (+ 287us) client_negotiation.cc:240] Negotiation successful
0926 07:34:06.628908 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":108,"threads_started":1}
WARNING: no policy specified for kudu/127.22.53.195@KRBTEST.COM; defaulting to no policy
I20260926 07:34:06.631516 8716 heartbeater.cc:347] Connected to a master server at 127.22.53.254:38243
I20260926 07:34:06.631803 8716 heartbeater.cc:464] Registering TS with master...
I20260926 07:34:06.632344 8716 heartbeater.cc:511] Master 127.22.53.254:38243 requested a full tablet report, sending...
I20260926 07:34:06.633682 8391 ts_manager.cc:194] Registered new tserver with Master: 09f9b7b30d5c46ea94ec98aa03d1f6bb (127.22.53.194:41173)
I20260926 07:34:06.634370 8391 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.53.194@KRBTEST.COM'} at 127.22.53.194:48183
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.53.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.53.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.195.keytab.
Entry for principal kudu/127.22.53.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.195.keytab.
I20260926 07:34:06.651965 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407783529717-22743-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407783529717-22743-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407783529717-22743-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407783529717-22743-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.53.195:0
--local_ip_for_outbound_sockets=127.22.53.195
--webserver_interface=127.22.53.195
--webserver_port=0
--tserver_master_addrs=127.22.53.254:38243
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.195.keytab
--principal=kudu/127.22.53.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:44991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:34:06.769276 8721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:34:06.769594 8721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:34:06.769649 8721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:34:06.773859 8721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:34:06.773948 8721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:34:06.774012 8721 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.53.195
I20260926 07:34:06.779827 8721 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:44991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407783529717-22743-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407783529717-22743-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.195.keytab
--principal=kudu/127.22.53.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.53.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407783529717-22743-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.53.195
--webserver_port=0
--tserver_master_addrs=127.22.53.254:38243
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8721
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.53.195
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407783529717-22743-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:34:06.781008 8721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:34:06.782042 8721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:34:06.792764 8729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:06.792833 8727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:06.792833 8726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:34:06.792961 8721 server_base.cc:1034] running on GCE node
I20260926 07:34:06.793514 8721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:34:06.794198 8721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:34:06.795377 8721 hybrid_clock.cc:648] HybridClock initialized: now 1790408046795328 us; error 62 us; skew 500 ppm
Sep 26 07:34:06 dist-test-slave-wdjc krb5kdc[8346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408046, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:34:06.798699 8721 init.cc:377] Logged in from keytab as kudu/127.22.53.195@KRBTEST.COM (short username kudu)
I20260926 07:34:06.799813 8721 webserver.cc:460] Webserver started at http://127.22.53.195:44615/ using document root <none> and password file <none>
I20260926 07:34:06.800446 8721 fs_manager.cc:359] Metadata directory not provided
I20260926 07:34:06.800513 8721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:34:06.800781 8721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:34:06.802983 8721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407783529717-22743-0/minicluster-data/ts-2/data/instance:
uuid: "cfb48e9c135c4a36a0947b66ada0af02"
format_stamp: "Formatted at 2026-09-26 07:34:06 on dist-test-slave-wdjc"
server_key: "3f60176d6a2bc47bbb82f37a8f44dfbe"
server_key_iv: "8e7aaeecd18d2b82722694b2032b8669"
server_key_version: "encryptionkey@0"
I20260926 07:34:06.803618 8721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407783529717-22743-0/minicluster-data/ts-2/wal/instance:
uuid: "cfb48e9c135c4a36a0947b66ada0af02"
format_stamp: "Formatted at 2026-09-26 07:34:06 on dist-test-slave-wdjc"
server_key: "3f60176d6a2bc47bbb82f37a8f44dfbe"
server_key_iv: "8e7aaeecd18d2b82722694b2032b8669"
server_key_version: "encryptionkey@0"
I20260926 07:34:06.807785 8721 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260926 07:34:06.810717 8736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:06.812134 8721 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 07:34:06.812255 8721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407783529717-22743-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407783529717-22743-0/minicluster-data/ts-2/wal
uuid: "cfb48e9c135c4a36a0947b66ada0af02"
format_stamp: "Formatted at 2026-09-26 07:34:06 on dist-test-slave-wdjc"
server_key: "3f60176d6a2bc47bbb82f37a8f44dfbe"
server_key_iv: "8e7aaeecd18d2b82722694b2032b8669"
server_key_version: "encryptionkey@0"
I20260926 07:34:06.812414 8721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407783529717-22743-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407783529717-22743-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407783529717-22743-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:34:06.827039 8721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:34:06.830552 8721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:34:06.830793 8721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:34:06.831444 8721 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:34:06.832315 8721 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:34:06.832389 8721 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:06.832463 8721 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:34:06.832508 8721 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:06.842460 8721 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.195:44561
I20260926 07:34:06.842489 8849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.195:44561 every 8 connection(s)
I20260926 07:34:06.843442 8721 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407783529717-22743-0/minicluster-data/ts-2/data/info.pb
I20260926 07:34:06.849233 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 8721
I20260926 07:34:06.849409 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407783529717-22743-0/minicluster-data/ts-2/wal/instance
I20260926 07:34:06.849720 22743 external_mini_cluster.cc:1428] Setting key 154a3d474001ee5191a8d950a56ef594
Sep 26 07:34:06 dist-test-slave-wdjc krb5kdc[8346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408046, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.195@KRBTEST.COM for kudu/127.22.53.254@KRBTEST.COM
I20260926 07:34:06.857749 8430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:06.845267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.195:34761 (local address 127.22.53.254:38243)
0926 07:34:06.845414 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:06.845418 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:06.846230 (+ 812us) server_negotiation.cc:407] Connection header received
0926 07:34:06.847634 (+ 1404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:06.847638 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:06.847702 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:06.847805 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:06.849059 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:06.849696 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:06.851045 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:06.851303 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:06.851406 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:06.854210 (+ 2804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:06.854214 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:06.854217 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:06.854246 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:06.855602 (+ 1356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:06.856145 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:06.856147 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:06.856148 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:06.856221 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:06.856527 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:06.856588 (+ 61us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:06.856592 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:06.856791 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:06.856870 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:06.857478 (+ 608us) server_negotiation.cc:299] Negotiation successful
0926 07:34:06.857599 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 07:34:06.858639 8852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:06.845629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:38243 (local address 127.22.53.195:34761)
0926 07:34:06.846059 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:06.846100 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:06.847414 (+ 1314us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:06.847979 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:06.847986 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:06.848348 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:06.848908 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:06.848923 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:06.849825 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:06.849830 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:06.850926 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:06.850936 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:06.851471 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:06.851475 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:06.851646 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:06.852372 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:06.852389 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:06.854004 (+ 1615us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:34:06.855737 (+ 1733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:06.855743 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:06.855756 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:06.856050 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:06.856303 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:06.856305 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:06.856307 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:06.856447 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:06.856915 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:34:06.856920 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:34:06.857198 (+ 278us) client_negotiation.cc:769] Sending connection context
0926 07:34:06.857407 (+ 209us) client_negotiation.cc:240] Negotiation successful
0926 07:34:06.857711 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":101,"threads_started":1}
I20260926 07:34:06.859936 8850 heartbeater.cc:347] Connected to a master server at 127.22.53.254:38243
I20260926 07:34:06.860177 8850 heartbeater.cc:464] Registering TS with master...
I20260926 07:34:06.860772 8850 heartbeater.cc:511] Master 127.22.53.254:38243 requested a full tablet report, sending...
I20260926 07:34:06.862028 8391 ts_manager.cc:194] Registered new tserver with Master: cfb48e9c135c4a36a0947b66ada0af02 (127.22.53.195:44561)
I20260926 07:34:06.862721 8391 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.53.195@KRBTEST.COM'} at 127.22.53.195:34761
I20260926 07:34:06.864140 22743 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:34:06.874858 8430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:06.866287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51562 (local address 127.22.53.254:38243)
0926 07:34:06.866432 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:06.866437 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:06.866632 (+ 195us) server_negotiation.cc:407] Connection header received
0926 07:34:06.866777 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:06.866782 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:06.866845 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:06.866938 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:06.867743 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:06.868300 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:06.869243 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:06.869470 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:06.869584 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:06.871258 (+ 1674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:06.871264 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:06.871272 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:06.871301 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:06.872788 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:06.873369 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:06.873378 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:06.873379 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:06.873456 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:06.873795 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:06.873798 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:06.873800 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:06.874026 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:06.874197 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:06.874509 (+ 312us) server_negotiation.cc:299] Negotiation successful
0926 07:34:06.874707 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 07:34:06.878283 8390 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51562:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:34:06.881377 8390 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:34:06.896214 8862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:06.890532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.195:44561 (local address 127.0.0.1:52778)
0926 07:34:06.891344 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:06.891395 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:06.891526 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:06.892469 (+ 943us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:06.892471 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:06.892486 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:34:06.892637 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:06.892643 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:06.893714 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:06.893717 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:06.894932 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:06.894942 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:06.895810 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:06.895812 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:06.895897 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:06.895927 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 07:34:06.896003 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 07:34:06.896091 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":637,"thread_start_us":87,"threads_started":1}
I20260926 07:34:06.896389 8863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:06.890716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52778 (local address 127.22.53.195:44561)
0926 07:34:06.891828 (+ 1112us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:06.891842 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:06.891866 (+ 24us) server_negotiation.cc:407] Connection header received
0926 07:34:06.891939 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:06.891944 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:06.892155 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:06.892307 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:34:06.892752 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:06.893567 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:06.895054 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:06.895689 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:06.895802 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:06.895979 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:06.896032 (+ 53us) server_negotiation.cc:299] Negotiation successful
0926 07:34:06.896129 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1005,"thread_start_us":156,"threads_started":1}
I20260926 07:34:06.897437 8865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:06.891239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.194:41173 (local address 127.0.0.1:60698)
0926 07:34:06.892029 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:06.892040 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:06.892112 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:06.892784 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:06.892786 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:06.892808 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:34:06.892893 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:06.892898 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:06.894196 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:06.894199 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:06.895582 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:06.895593 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:06.897016 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:06.897020 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:06.897114 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:06.897139 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 07:34:06.897219 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 07:34:06.897298 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":727,"mutex_wait_us":185,"thread_start_us":84,"threads_started":1}
I20260926 07:34:06.897513 8866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:06.891650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60698 (local address 127.22.53.194:41173)
0926 07:34:06.892347 (+ 697us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:06.892358 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:06.892380 (+ 22us) server_negotiation.cc:407] Connection header received
0926 07:34:06.892450 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:06.892457 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:06.892668 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:06.892797 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:34:06.893013 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:06.894063 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:06.895728 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:06.896885 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:06.896994 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:06.897132 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:06.897269 (+ 137us) server_negotiation.cc:299] Negotiation successful
0926 07:34:06.897367 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":116,"threads_started":1}
I20260926 07:34:06.898988 8861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:06.890728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.193:41295 (local address 127.0.0.1:56980)
0926 07:34:06.891510 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:06.891523 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:06.891591 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:06.892317 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:06.892322 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:06.892345 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:34:06.892499 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:06.892506 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:06.894164 (+ 1658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:06.894170 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:06.895582 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:06.895593 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:06.898573 (+ 2980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:06.898577 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:06.898688 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:06.898710 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 07:34:06.898791 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 07:34:06.898858 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":726,"mutex_wait_us":56,"thread_start_us":106,"threads_started":1}
I20260926 07:34:06.899092 8864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:06.891049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56980 (local address 127.22.53.193:41295)
0926 07:34:06.891830 (+ 781us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:06.891844 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:06.891866 (+ 22us) server_negotiation.cc:407] Connection header received
0926 07:34:06.891939 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:06.891944 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:06.892150 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:06.892307 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:34:06.892663 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:06.893882 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:06.896601 (+ 2719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:06.897840 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:06.897956 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:06.898063 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:06.898853 (+ 790us) server_negotiation.cc:299] Negotiation successful
0926 07:34:06.898952 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":692,"thread_start_us":64,"threads_started":1}
I20260926 07:34:06.899711 8784 tablet_service.cc:1467] Processing CreateTablet for tablet aa97c07145824099a4750c98c603dc86 (DEFAULT_TABLE table=test-table [id=ef8f5b803489484fa35df226b932c733]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:34:06.900192 8650 tablet_service.cc:1467] Processing CreateTablet for tablet aa97c07145824099a4750c98c603dc86 (DEFAULT_TABLE table=test-table [id=ef8f5b803489484fa35df226b932c733]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:34:06.900848 8784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa97c07145824099a4750c98c603dc86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:34:06.901098 8650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa97c07145824099a4750c98c603dc86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:34:06.907137 8867 tablet_bootstrap.cc:492] T aa97c07145824099a4750c98c603dc86 P 09f9b7b30d5c46ea94ec98aa03d1f6bb: Bootstrap starting.
I20260926 07:34:06.906873 8516 tablet_service.cc:1467] Processing CreateTablet for tablet aa97c07145824099a4750c98c603dc86 (DEFAULT_TABLE table=test-table [id=ef8f5b803489484fa35df226b932c733]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:34:06.908044 8516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa97c07145824099a4750c98c603dc86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:34:06.908437 8868 tablet_bootstrap.cc:492] T aa97c07145824099a4750c98c603dc86 P cfb48e9c135c4a36a0947b66ada0af02: Bootstrap starting.
I20260926 07:34:06.910562 8868 tablet_bootstrap.cc:654] T aa97c07145824099a4750c98c603dc86 P cfb48e9c135c4a36a0947b66ada0af02: Neither blocks nor log segments found. Creating new log.
I20260926 07:34:06.910585 8867 tablet_bootstrap.cc:654] T aa97c07145824099a4750c98c603dc86 P 09f9b7b30d5c46ea94ec98aa03d1f6bb: Neither blocks nor log segments found. Creating new log.
I20260926 07:34:06.911923 8867 log.cc:824] T aa97c07145824099a4750c98c603dc86 P 09f9b7b30d5c46ea94ec98aa03d1f6bb: Log is configured to *not* fsync() on all Append() calls
I20260926 07:34:06.911926 8868 log.cc:824] T aa97c07145824099a4750c98c603dc86 P cfb48e9c135c4a36a0947b66ada0af02: Log is configured to *not* fsync() on all Append() calls
I20260926 07:34:06.914237 8867 tablet_bootstrap.cc:492] T aa97c07145824099a4750c98c603dc86 P 09f9b7b30d5c46ea94ec98aa03d1f6bb: No bootstrap required, opened a new log
I20260926 07:34:06.914469 8867 ts_tablet_manager.cc:1397] T aa97c07145824099a4750c98c603dc86 P 09f9b7b30d5c46ea94ec98aa03d1f6bb: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260926 07:34:06.915380 8868 tablet_bootstrap.cc:492] T aa97c07145824099a4750c98c603dc86 P cfb48e9c135c4a36a0947b66ada0af02: No bootstrap required, opened a new log
I20260926 07:34:06.915644 8868 ts_tablet_manager.cc:1397] T aa97c07145824099a4750c98c603dc86 P cfb48e9c135c4a36a0947b66ada0af02: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260926 07:34:06.916031 8871 tablet_bootstrap.cc:492] T aa97c07145824099a4750c98c603dc86 P aa3a909d1b614776a0a8f7b27cd0b35c: Bootstrap starting.
I20260926 07:34:06.918243 8867 raft_consensus.cc:348] T aa97c07145824099a4750c98c603dc86 P 09f9b7b30d5c46ea94ec98aa03d1f6bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfb48e9c135c4a36a0947b66ada0af02" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 44561 } } peers { permanent_uuid: "aa3a909d1b614776a0a8f7b27cd0b35c" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 41295 } } peers { permanent_uuid: "09f9b7b30d5c46ea94ec98aa03d1f6bb" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 41173 } }
I20260926 07:34:06.918524 8867 raft_consensus.cc:374] T aa97c07145824099a4750c98c603dc86 P 09f9b7b30d5c46ea94ec98aa03d1f6bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:34:06.918582 8867 raft_consensus.cc:729] T aa97c07145824099a4750c98c603dc86 P 09f9b7b30d5c46ea94ec98aa03d1f6bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09f9b7b30d5c46ea94ec98aa03d1f6bb, State: Initialized, Role: FOLLOWER
I20260926 07:34:06.918889 8871 tablet_bootstrap.cc:654] T aa97c07145824099a4750c98c603dc86 P aa3a909d1b614776a0a8f7b27cd0b35c: Neither blocks nor log segments found. Creating new log.
I20260926 07:34:06.919092 8867 consensus_queue.cc:260] T aa97c07145824099a4750c98c603dc86 P 09f9b7b30d5c46ea94ec98aa03d1f6bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfb48e9c135c4a36a0947b66ada0af02" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 44561 } } peers { permanent_uuid: "aa3a909d1b614776a0a8f7b27cd0b35c" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 41295 } } peers { permanent_uuid: "09f9b7b30d5c46ea94ec98aa03d1f6bb" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 41173 } }
I20260926 07:34:06.919625 8868 raft_consensus.cc:348] T aa97c07145824099a4750c98c603dc86 P cfb48e9c135c4a36a0947b66ada0af02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfb48e9c135c4a36a0947b66ada0af02" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 44561 } } peers { permanent_uuid: "aa3a909d1b614776a0a8f7b27cd0b35c" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 41295 } } peers { permanent_uuid: "09f9b7b30d5c46ea94ec98aa03d1f6bb" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 41173 } }
I20260926 07:34:06.919845 8716 heartbeater.cc:503] Master 127.22.53.254:38243 was elected leader, sending a full tablet report...
I20260926 07:34:06.919858 8871 log.cc:824] T aa97c07145824099a4750c98c603dc86 P aa3a909d1b614776a0a8f7b27cd0b35c: Log is configured to *not* fsync() on all Append() calls
I20260926 07:34:06.919914 8868 raft_consensus.cc:374] T aa97c07145824099a4750c98c603dc86 P cfb48e9c135c4a36a0947b66ada0af02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:34:06.919961 8868 raft_consensus.cc:729] T aa97c07145824099a4750c98c603dc86 P cfb48e9c135c4a36a0947b66ada0af02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfb48e9c135c4a36a0947b66ada0af02, State: Initialized, Role: FOLLOWER
I20260926 07:34:06.920195 8867 ts_tablet_manager.cc:1428] T aa97c07145824099a4750c98c603dc86 P 09f9b7b30d5c46ea94ec98aa03d1f6bb: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260926 07:34:06.920711 8868 consensus_queue.cc:260] T aa97c07145824099a4750c98c603dc86 P cfb48e9c135c4a36a0947b66ada0af02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfb48e9c135c4a36a0947b66ada0af02" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 44561 } } peers { permanent_uuid: "aa3a909d1b614776a0a8f7b27cd0b35c" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 41295 } } peers { permanent_uuid: "09f9b7b30d5c46ea94ec98aa03d1f6bb" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 41173 } }
I20260926 07:34:06.921633 8868 ts_tablet_manager.cc:1428] T aa97c07145824099a4750c98c603dc86 P cfb48e9c135c4a36a0947b66ada0af02: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260926 07:34:06.922042 8850 heartbeater.cc:503] Master 127.22.53.254:38243 was elected leader, sending a full tablet report...
I20260926 07:34:06.922646 8871 tablet_bootstrap.cc:492] T aa97c07145824099a4750c98c603dc86 P aa3a909d1b614776a0a8f7b27cd0b35c: No bootstrap required, opened a new log
I20260926 07:34:06.922849 8871 ts_tablet_manager.cc:1397] T aa97c07145824099a4750c98c603dc86 P aa3a909d1b614776a0a8f7b27cd0b35c: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260926 07:34:06.926784 8871 raft_consensus.cc:348] T aa97c07145824099a4750c98c603dc86 P aa3a909d1b614776a0a8f7b27cd0b35c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfb48e9c135c4a36a0947b66ada0af02" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 44561 } } peers { permanent_uuid: "aa3a909d1b614776a0a8f7b27cd0b35c" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 41295 } } peers { permanent_uuid: "09f9b7b30d5c46ea94ec98aa03d1f6bb" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 41173 } }
I20260926 07:34:06.927048 8871 raft_consensus.cc:374] T aa97c07145824099a4750c98c603dc86 P aa3a909d1b614776a0a8f7b27cd0b35c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:34:06.927111 8871 raft_consensus.cc:729] T aa97c07145824099a4750c98c603dc86 P aa3a909d1b614776a0a8f7b27cd0b35c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa3a909d1b614776a0a8f7b27cd0b35c, State: Initialized, Role: FOLLOWER
I20260926 07:34:06.927546 8871 consensus_queue.cc:260] T aa97c07145824099a4750c98c603dc86 P aa3a909d1b614776a0a8f7b27cd0b35c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfb48e9c135c4a36a0947b66ada0af02" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 44561 } } peers { permanent_uuid: "aa3a909d1b614776a0a8f7b27cd0b35c" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 41295 } } peers { permanent_uuid: "09f9b7b30d5c46ea94ec98aa03d1f6bb" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 41173 } }
I20260926 07:34:06.928339 8582 heartbeater.cc:503] Master 127.22.53.254:38243 was elected leader, sending a full tablet report...
I20260926 07:34:06.928696 8871 ts_tablet_manager.cc:1428] T aa97c07145824099a4750c98c603dc86 P aa3a909d1b614776a0a8f7b27cd0b35c: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260926 07:34:06.949265 8875 raft_consensus.cc:482] T aa97c07145824099a4750c98c603dc86 P aa3a909d1b614776a0a8f7b27cd0b35c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:34:06.949517 8875 raft_consensus.cc:504] T aa97c07145824099a4750c98c603dc86 P aa3a909d1b614776a0a8f7b27cd0b35c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfb48e9c135c4a36a0947b66ada0af02" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 44561 } } peers { permanent_uuid: "aa3a909d1b614776a0a8f7b27cd0b35c" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 41295 } } peers { permanent_uuid: "09f9b7b30d5c46ea94ec98aa03d1f6bb" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 41173 } }
I20260926 07:34:06.950824 8875 leader_election.cc:288] T aa97c07145824099a4750c98c603dc86 P aa3a909d1b614776a0a8f7b27cd0b35c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cfb48e9c135c4a36a0947b66ada0af02 (127.22.53.195:44561), 09f9b7b30d5c46ea94ec98aa03d1f6bb (127.22.53.194:41173)
I20260926 07:34:06.955238 8866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:06.951321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.193:48491 (local address 127.22.53.194:41173)
0926 07:34:06.951472 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:06.951477 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:06.951665 (+ 188us) server_negotiation.cc:407] Connection header received
0926 07:34:06.951871 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:06.951874 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:06.951919 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:06.952009 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:34:06.952406 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:06.952998 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:06.954184 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:06.954612 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:06.954761 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:06.954830 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:06.955014 (+ 184us) server_negotiation.cc:299] Negotiation successful
0926 07:34:06.955089 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 07:34:06.955283 8877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:06.951153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.194:41173 (local address 127.22.53.193:48491)
0926 07:34:06.951562 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:06.951574 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:06.951735 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:06.952023 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:06.952026 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:06.952054 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:34:06.952276 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:06.952282 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:06.953123 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:06.953129 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:06.954038 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:06.954048 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:06.954756 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:06.954760 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:06.954864 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:06.954902 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 07:34:06.954993 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 07:34:06.955075 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"mutex_wait_us":68,"spinlock_wait_cycles":24320,"thread_start_us":125,"threads_started":1}
I20260926 07:34:06.955298 8863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:06.951311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.193:54987 (local address 127.22.53.195:44561)
0926 07:34:06.951462 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:06.951468 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:06.951642 (+ 174us) server_negotiation.cc:407] Connection header received
0926 07:34:06.951851 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:06.951857 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:06.951914 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:06.952017 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:34:06.952392 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:06.953000 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:06.954215 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:06.954649 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:06.954805 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:06.954868 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:06.955080 (+ 212us) server_negotiation.cc:299] Negotiation successful
0926 07:34:06.955162 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 07:34:06.955761 8876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:06.951059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.195:44561 (local address 127.22.53.193:54987)
0926 07:34:06.951534 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:06.951547 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:06.951730 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:06.952026 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:06.952029 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:06.952060 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:34:06.952262 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:06.952269 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:06.953138 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:06.953142 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:06.954059 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:06.954069 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:06.954795 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:06.954799 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:06.954942 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:06.954963 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 07:34:06.955063 (+ 100us) client_negotiation.cc:240] Negotiation successful
0926 07:34:06.955155 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396,"mutex_wait_us":84,"spinlock_wait_cycles":30976,"thread_start_us":150,"threads_started":1}
I20260926 07:34:06.956063 8670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa97c07145824099a4750c98c603dc86" candidate_uuid: "aa3a909d1b614776a0a8f7b27cd0b35c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09f9b7b30d5c46ea94ec98aa03d1f6bb" is_pre_election: true
I20260926 07:34:06.956422 8670 raft_consensus.cc:2454] T aa97c07145824099a4750c98c603dc86 P 09f9b7b30d5c46ea94ec98aa03d1f6bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa3a909d1b614776a0a8f7b27cd0b35c in term 0.
I20260926 07:34:06.956558 8804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa97c07145824099a4750c98c603dc86" candidate_uuid: "aa3a909d1b614776a0a8f7b27cd0b35c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfb48e9c135c4a36a0947b66ada0af02" is_pre_election: true
I20260926 07:34:06.956938 8804 raft_consensus.cc:2454] T aa97c07145824099a4750c98c603dc86 P cfb48e9c135c4a36a0947b66ada0af02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa3a909d1b614776a0a8f7b27cd0b35c in term 0.
I20260926 07:34:06.957010 8471 leader_election.cc:302] T aa97c07145824099a4750c98c603dc86 P aa3a909d1b614776a0a8f7b27cd0b35c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09f9b7b30d5c46ea94ec98aa03d1f6bb, aa3a909d1b614776a0a8f7b27cd0b35c; no voters:
I20260926 07:34:06.957383 8875 raft_consensus.cc:2781] T aa97c07145824099a4750c98c603dc86 P aa3a909d1b614776a0a8f7b27cd0b35c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:34:06.957499 8875 raft_consensus.cc:482] T aa97c07145824099a4750c98c603dc86 P aa3a909d1b614776a0a8f7b27cd0b35c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:34:06.957545 8875 raft_consensus.cc:3037] T aa97c07145824099a4750c98c603dc86 P aa3a909d1b614776a0a8f7b27cd0b35c [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:34:06.958657 8875 raft_consensus.cc:504] T aa97c07145824099a4750c98c603dc86 P aa3a909d1b614776a0a8f7b27cd0b35c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfb48e9c135c4a36a0947b66ada0af02" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 44561 } } peers { permanent_uuid: "aa3a909d1b614776a0a8f7b27cd0b35c" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 41295 } } peers { permanent_uuid: "09f9b7b30d5c46ea94ec98aa03d1f6bb" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 41173 } }
I20260926 07:34:06.959038 8875 leader_election.cc:288] T aa97c07145824099a4750c98c603dc86 P aa3a909d1b614776a0a8f7b27cd0b35c [CANDIDATE]: Term 1 election: Requested vote from peers cfb48e9c135c4a36a0947b66ada0af02 (127.22.53.195:44561), 09f9b7b30d5c46ea94ec98aa03d1f6bb (127.22.53.194:41173)
I20260926 07:34:06.959456 8804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa97c07145824099a4750c98c603dc86" candidate_uuid: "aa3a909d1b614776a0a8f7b27cd0b35c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfb48e9c135c4a36a0947b66ada0af02"
I20260926 07:34:06.959563 8670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa97c07145824099a4750c98c603dc86" candidate_uuid: "aa3a909d1b614776a0a8f7b27cd0b35c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09f9b7b30d5c46ea94ec98aa03d1f6bb"
I20260926 07:34:06.959616 8804 raft_consensus.cc:3037] T aa97c07145824099a4750c98c603dc86 P cfb48e9c135c4a36a0947b66ada0af02 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:34:06.959733 8670 raft_consensus.cc:3037] T aa97c07145824099a4750c98c603dc86 P 09f9b7b30d5c46ea94ec98aa03d1f6bb [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:34:06.961097 8670 raft_consensus.cc:2454] T aa97c07145824099a4750c98c603dc86 P 09f9b7b30d5c46ea94ec98aa03d1f6bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa3a909d1b614776a0a8f7b27cd0b35c in term 1.
I20260926 07:34:06.961210 8804 raft_consensus.cc:2454] T aa97c07145824099a4750c98c603dc86 P cfb48e9c135c4a36a0947b66ada0af02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa3a909d1b614776a0a8f7b27cd0b35c in term 1.
I20260926 07:34:06.961488 8471 leader_election.cc:302] T aa97c07145824099a4750c98c603dc86 P aa3a909d1b614776a0a8f7b27cd0b35c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09f9b7b30d5c46ea94ec98aa03d1f6bb, aa3a909d1b614776a0a8f7b27cd0b35c; no voters:
I20260926 07:34:06.961742 8875 raft_consensus.cc:2781] T aa97c07145824099a4750c98c603dc86 P aa3a909d1b614776a0a8f7b27cd0b35c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:34:06.961964 8875 raft_consensus.cc:686] T aa97c07145824099a4750c98c603dc86 P aa3a909d1b614776a0a8f7b27cd0b35c [term 1 LEADER]: Becoming Leader. State: Replica: aa3a909d1b614776a0a8f7b27cd0b35c, State: Running, Role: LEADER
I20260926 07:34:06.962386 8875 consensus_queue.cc:237] T aa97c07145824099a4750c98c603dc86 P aa3a909d1b614776a0a8f7b27cd0b35c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfb48e9c135c4a36a0947b66ada0af02" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 44561 } } peers { permanent_uuid: "aa3a909d1b614776a0a8f7b27cd0b35c" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 41295 } } peers { permanent_uuid: "09f9b7b30d5c46ea94ec98aa03d1f6bb" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 41173 } }
I20260926 07:34:06.965919 8390 catalog_manager.cc:5495] T aa97c07145824099a4750c98c603dc86 P aa3a909d1b614776a0a8f7b27cd0b35c reported cstate change: term changed from 0 to 1, leader changed from <none> to aa3a909d1b614776a0a8f7b27cd0b35c (127.22.53.193). New cstate: current_term: 1 leader_uuid: "aa3a909d1b614776a0a8f7b27cd0b35c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfb48e9c135c4a36a0947b66ada0af02" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 44561 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aa3a909d1b614776a0a8f7b27cd0b35c" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 41295 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09f9b7b30d5c46ea94ec98aa03d1f6bb" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 41173 } health_report { overall_health: UNKNOWN } } }
I20260926 07:34:06.987082 8864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:06.983039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56988 (local address 127.22.53.193:41295)
0926 07:34:06.983182 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:06.983186 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:06.983202 (+ 16us) server_negotiation.cc:407] Connection header received
0926 07:34:06.983333 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:06.983335 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:06.983392 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:06.983487 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:34:06.983842 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:06.984346 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:06.985380 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:06.985651 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:06.985784 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:06.986059 (+ 275us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:34:06.986524 (+ 465us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:34:06.986641 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:06.986845 (+ 204us) server_negotiation.cc:299] Negotiation successful
0926 07:34:06.986906 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 07:34:06.993535 8670 raft_consensus.cc:1261] T aa97c07145824099a4750c98c603dc86 P 09f9b7b30d5c46ea94ec98aa03d1f6bb [term 1 FOLLOWER]: Refusing update from remote peer aa3a909d1b614776a0a8f7b27cd0b35c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:34:06.993535 8804 raft_consensus.cc:1261] T aa97c07145824099a4750c98c603dc86 P cfb48e9c135c4a36a0947b66ada0af02 [term 1 FOLLOWER]: Refusing update from remote peer aa3a909d1b614776a0a8f7b27cd0b35c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:34:06.994285 8875 consensus_queue.cc:1035] T aa97c07145824099a4750c98c603dc86 P aa3a909d1b614776a0a8f7b27cd0b35c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfb48e9c135c4a36a0947b66ada0af02" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 44561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:34:06.994526 8878 consensus_queue.cc:1035] T aa97c07145824099a4750c98c603dc86 P aa3a909d1b614776a0a8f7b27cd0b35c [LEADER]: Connected to new peer: Peer: permanent_uuid: "09f9b7b30d5c46ea94ec98aa03d1f6bb" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 41173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:34:07.015066 8391 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:34:07.015257 8391 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:34:07.018637 8391 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0d614df8bdf94a518700fd14f0687e77: Sending DeleteTablet for 3 replicas of tablet aa97c07145824099a4750c98c603dc86
I20260926 07:34:07.019721 8650 tablet_service.cc:1514] Processing DeleteTablet for tablet aa97c07145824099a4750c98c603dc86 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:34:07 UTC) from {username='kudu', principal='kudu/127.22.53.254@KRBTEST.COM'} at 127.0.0.1:60698
I20260926 07:34:07.019727 8516 tablet_service.cc:1514] Processing DeleteTablet for tablet aa97c07145824099a4750c98c603dc86 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:34:07 UTC) from {username='kudu', principal='kudu/127.22.53.254@KRBTEST.COM'} at 127.0.0.1:56980
I20260926 07:34:07.019737 8784 tablet_service.cc:1514] Processing DeleteTablet for tablet aa97c07145824099a4750c98c603dc86 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:34:07 UTC) from {username='kudu', principal='kudu/127.22.53.254@KRBTEST.COM'} at 127.0.0.1:52778
I20260926 07:34:07.020334 8891 tablet_replica.cc:330] T aa97c07145824099a4750c98c603dc86 P 09f9b7b30d5c46ea94ec98aa03d1f6bb: stopping tablet replica
I20260926 07:34:07.020380 8893 tablet_replica.cc:330] T aa97c07145824099a4750c98c603dc86 P cfb48e9c135c4a36a0947b66ada0af02: stopping tablet replica
I20260926 07:34:07.020798 8893 raft_consensus.cc:2229] T aa97c07145824099a4750c98c603dc86 P cfb48e9c135c4a36a0947b66ada0af02 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:34:07.020798 8891 raft_consensus.cc:2229] T aa97c07145824099a4750c98c603dc86 P 09f9b7b30d5c46ea94ec98aa03d1f6bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:34:07.021101 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 8453
I20260926 07:34:07.021673 8891 raft_consensus.cc:2258] T aa97c07145824099a4750c98c603dc86 P 09f9b7b30d5c46ea94ec98aa03d1f6bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:34:07.021202 8893 raft_consensus.cc:2258] T aa97c07145824099a4750c98c603dc86 P cfb48e9c135c4a36a0947b66ada0af02 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:34:07.023600 8891 ts_tablet_manager.cc:1905] T aa97c07145824099a4750c98c603dc86 P 09f9b7b30d5c46ea94ec98aa03d1f6bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:34:07.024591 8893 ts_tablet_manager.cc:1905] T aa97c07145824099a4750c98c603dc86 P cfb48e9c135c4a36a0947b66ada0af02: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:34:07.026679 8891 ts_tablet_manager.cc:1918] T aa97c07145824099a4750c98c603dc86 P 09f9b7b30d5c46ea94ec98aa03d1f6bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:34:07.026784 8891 log.cc:1196] T aa97c07145824099a4750c98c603dc86 P 09f9b7b30d5c46ea94ec98aa03d1f6bb: Deleting WAL directory at /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407783529717-22743-0/minicluster-data/ts-1/wal/wals/aa97c07145824099a4750c98c603dc86
I20260926 07:34:07.027112 8891 ts_tablet_manager.cc:1939] T aa97c07145824099a4750c98c603dc86 P 09f9b7b30d5c46ea94ec98aa03d1f6bb: Deleting consensus metadata
I20260926 07:34:07.027112 8893 ts_tablet_manager.cc:1918] T aa97c07145824099a4750c98c603dc86 P cfb48e9c135c4a36a0947b66ada0af02: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:34:07.027221 8893 log.cc:1196] T aa97c07145824099a4750c98c603dc86 P cfb48e9c135c4a36a0947b66ada0af02: Deleting WAL directory at /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407783529717-22743-0/minicluster-data/ts-2/wal/wals/aa97c07145824099a4750c98c603dc86
I20260926 07:34:07.027546 8893 ts_tablet_manager.cc:1939] T aa97c07145824099a4750c98c603dc86 P cfb48e9c135c4a36a0947b66ada0af02: Deleting consensus metadata
I20260926 07:34:07.028482 8377 catalog_manager.cc:4841] TS 09f9b7b30d5c46ea94ec98aa03d1f6bb (127.22.53.194:41173): tablet aa97c07145824099a4750c98c603dc86 (table test-table [id=ef8f5b803489484fa35df226b932c733]) successfully deleted
I20260926 07:34:07.028650 8378 catalog_manager.cc:4841] TS cfb48e9c135c4a36a0947b66ada0af02 (127.22.53.195:44561): tablet aa97c07145824099a4750c98c603dc86 (table test-table [id=ef8f5b803489484fa35df226b932c733]) successfully deleted
W20260926 07:34:07.033272 8377 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.53.193:41295 (error 108)
I20260926 07:34:07.033360 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 8587
I20260926 07:34:07.035584 8861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:07.035122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.193:41295 (local address 127.0.0.1:56992)
0926 07:34:07.035381 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:07.035465 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.193:41295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142}
W20260926 07:34:07.035925 8377 catalog_manager.cc:4584] TS aa3a909d1b614776a0a8f7b27cd0b35c (127.22.53.193:41295): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet aa97c07145824099a4750c98c603dc86: Network error: Client connection negotiation failed: client connection to 127.22.53.193:41295: connect: Connection refused (error 111)
I20260926 07:34:07.040887 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 8721
I20260926 07:34:07.046850 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 8359
2026-09-26T07:34:07Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2402 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-26T07:34:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:34:07Z Disabled control of system clock
I20260926 07:34:07.079196 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407783529717-22743-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407783529717-22743-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407783529717-22743-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.53.254:36349
--webserver_interface=127.22.53.254
--webserver_port=0
--builtin_ntp_servers=127.22.53.212:45565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.53.254:36349
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 07:34:07.195037 8898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:34:07.195350 8898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:34:07.195389 8898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:34:07.199608 8898 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:34:07.199697 8898 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:34:07.199712 8898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:34:07.199731 8898 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:34:07.199747 8898 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:34:07.204988 8898 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:45565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407783529717-22743-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407783529717-22743-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.53.254:36349
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.53.254:36349
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.53.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8898
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407783529717-22743-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:34:07.206144 8898 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:34:07.207528 8898 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:34:07.217172 8903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:07.217223 8904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:34:07.217200 8898 server_base.cc:1034] running on GCE node
W20260926 07:34:07.217172 8906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:34:07.218096 8898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:34:07.219244 8898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:34:07.220467 8898 hybrid_clock.cc:648] HybridClock initialized: now 1790408047220455 us; error 31 us; skew 500 ppm
I20260926 07:34:07.222748 8898 webserver.cc:460] Webserver started at http://127.22.53.254:41245/ using document root <none> and password file <none>
I20260926 07:34:07.223423 8898 fs_manager.cc:359] Metadata directory not provided
I20260926 07:34:07.223528 8898 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:34:07.223774 8898 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:34:07.226186 8898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407783529717-22743-0/minicluster-data/master-0/data/instance:
uuid: "a3cf74af4a654b9998bc15bf8d4d2a74"
format_stamp: "Formatted at 2026-09-26 07:34:07 on dist-test-slave-wdjc"
server_key: "6738866ea743982395915cdb9a360ef9"
server_key_iv: "8125f598121372cf8ab92ae980b433e1"
server_key_version: "encryptionkey@0"
I20260926 07:34:07.226821 8898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407783529717-22743-0/minicluster-data/master-0/wal/instance:
uuid: "a3cf74af4a654b9998bc15bf8d4d2a74"
format_stamp: "Formatted at 2026-09-26 07:34:07 on dist-test-slave-wdjc"
server_key: "6738866ea743982395915cdb9a360ef9"
server_key_iv: "8125f598121372cf8ab92ae980b433e1"
server_key_version: "encryptionkey@0"
I20260926 07:34:07.230978 8898 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260926 07:34:07.233882 8912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:07.235576 8898 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:34:07.235741 8898 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407783529717-22743-0/minicluster-data/master-0/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407783529717-22743-0/minicluster-data/master-0/wal
uuid: "a3cf74af4a654b9998bc15bf8d4d2a74"
format_stamp: "Formatted at 2026-09-26 07:34:07 on dist-test-slave-wdjc"
server_key: "6738866ea743982395915cdb9a360ef9"
server_key_iv: "8125f598121372cf8ab92ae980b433e1"
server_key_version: "encryptionkey@0"
I20260926 07:34:07.235901 8898 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407783529717-22743-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407783529717-22743-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407783529717-22743-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:34:07.252998 8898 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:34:07.253805 8898 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:34:07.254027 8898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:34:07.261518 8898 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.254:36349
I20260926 07:34:07.261540 8964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.254:36349 every 8 connection(s)
I20260926 07:34:07.262641 8898 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
I20260926 07:34:07.266050 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 8898
I20260926 07:34:07.266127 8965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:34:07.266192 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407783529717-22743-0/minicluster-data/master-0/wal/instance
I20260926 07:34:07.266547 22743 external_mini_cluster.cc:1428] Setting key 4d12ac448d69b209bfbb76f1b01c24d3
I20260926 07:34:07.273806 8965 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3cf74af4a654b9998bc15bf8d4d2a74: Bootstrap starting.
I20260926 07:34:07.276710 8965 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a3cf74af4a654b9998bc15bf8d4d2a74: Neither blocks nor log segments found. Creating new log.
I20260926 07:34:07.277144 8968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:07.268139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50320 (local address 127.22.53.254:36349)
0926 07:34:07.268851 (+ 712us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:07.268867 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:07.268907 (+ 40us) server_negotiation.cc:407] Connection header received
0926 07:34:07.269626 (+ 719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:07.269651 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:07.270152 (+ 501us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:07.270467 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:07.270887 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:07.271871 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:07.273134 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:07.273839 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:07.274044 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:07.275052 (+ 1008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:07.275058 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:07.275073 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:34:07.275509 (+ 436us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:07.275572 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:34:07.275594 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:07.275694 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:07.275874 (+ 180us) server_negotiation.cc:299] Negotiation successful
0926 07:34:07.276201 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":168,"threads_started":1}
I20260926 07:34:07.277624 8965 log.cc:824] T 00000000000000000000000000000000 P a3cf74af4a654b9998bc15bf8d4d2a74: Log is configured to *not* fsync() on all Append() calls
I20260926 07:34:07.279714 8965 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3cf74af4a654b9998bc15bf8d4d2a74: No bootstrap required, opened a new log
I20260926 07:34:07.283691 8965 raft_consensus.cc:348] T 00000000000000000000000000000000 P a3cf74af4a654b9998bc15bf8d4d2a74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3cf74af4a654b9998bc15bf8d4d2a74" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 36349 } }
I20260926 07:34:07.284032 8965 raft_consensus.cc:374] T 00000000000000000000000000000000 P a3cf74af4a654b9998bc15bf8d4d2a74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:34:07.284125 8965 raft_consensus.cc:729] T 00000000000000000000000000000000 P a3cf74af4a654b9998bc15bf8d4d2a74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3cf74af4a654b9998bc15bf8d4d2a74, State: Initialized, Role: FOLLOWER
I20260926 07:34:07.284797 8965 consensus_queue.cc:260] T 00000000000000000000000000000000 P a3cf74af4a654b9998bc15bf8d4d2a74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3cf74af4a654b9998bc15bf8d4d2a74" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 36349 } }
I20260926 07:34:07.285005 8965 raft_consensus.cc:388] T 00000000000000000000000000000000 P a3cf74af4a654b9998bc15bf8d4d2a74 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:34:07.285092 8965 raft_consensus.cc:482] T 00000000000000000000000000000000 P a3cf74af4a654b9998bc15bf8d4d2a74 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:34:07.285238 8965 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a3cf74af4a654b9998bc15bf8d4d2a74 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:34:07.286371 8965 raft_consensus.cc:504] T 00000000000000000000000000000000 P a3cf74af4a654b9998bc15bf8d4d2a74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3cf74af4a654b9998bc15bf8d4d2a74" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 36349 } }
I20260926 07:34:07.286783 8965 leader_election.cc:302] T 00000000000000000000000000000000 P a3cf74af4a654b9998bc15bf8d4d2a74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a3cf74af4a654b9998bc15bf8d4d2a74; no voters:
I20260926 07:34:07.287159 8965 leader_election.cc:288] T 00000000000000000000000000000000 P a3cf74af4a654b9998bc15bf8d4d2a74 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:34:07.287372 8970 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a3cf74af4a654b9998bc15bf8d4d2a74 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:34:07.287637 8970 raft_consensus.cc:686] T 00000000000000000000000000000000 P a3cf74af4a654b9998bc15bf8d4d2a74 [term 1 LEADER]: Becoming Leader. State: Replica: a3cf74af4a654b9998bc15bf8d4d2a74, State: Running, Role: LEADER
I20260926 07:34:07.288416 8970 consensus_queue.cc:237] T 00000000000000000000000000000000 P a3cf74af4a654b9998bc15bf8d4d2a74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3cf74af4a654b9998bc15bf8d4d2a74" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 36349 } }
I20260926 07:34:07.288611 8965 sys_catalog.cc:564] T 00000000000000000000000000000000 P a3cf74af4a654b9998bc15bf8d4d2a74 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:34:07.290102 8971 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3cf74af4a654b9998bc15bf8d4d2a74 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a3cf74af4a654b9998bc15bf8d4d2a74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3cf74af4a654b9998bc15bf8d4d2a74" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 36349 } } }
I20260926 07:34:07.290256 8971 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3cf74af4a654b9998bc15bf8d4d2a74 [sys.catalog]: This master's current role is: LEADER
I20260926 07:34:07.290683 8981 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:34:07.290637 8972 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3cf74af4a654b9998bc15bf8d4d2a74 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a3cf74af4a654b9998bc15bf8d4d2a74. Latest consensus state: current_term: 1 leader_uuid: "a3cf74af4a654b9998bc15bf8d4d2a74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3cf74af4a654b9998bc15bf8d4d2a74" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 36349 } } }
I20260926 07:34:07.290782 8972 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3cf74af4a654b9998bc15bf8d4d2a74 [sys.catalog]: This master's current role is: LEADER
I20260926 07:34:07.294385 8981 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:34:07.301939 8981 catalog_manager.cc:1347] Generated new cluster ID: 50961eb85edb4c8b8b7315b0956f8fe5
I20260926 07:34:07.302034 8981 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:34:07.335117 8981 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:34:07.336202 8981 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:34:07.351114 8981 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a3cf74af4a654b9998bc15bf8d4d2a74: Generated new TSK 0
I20260926 07:34:07.352161 8981 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:34:07.360244 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407783529717-22743-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407783529717-22743-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407783529717-22743-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407783529717-22743-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.53.193:0
--local_ip_for_outbound_sockets=127.22.53.193
--webserver_interface=127.22.53.193
--webserver_port=0
--tserver_master_addrs=127.22.53.254:36349
--builtin_ntp_servers=127.22.53.212:45565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 07:34:07.534668 8989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:34:07.535055 8989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:34:07.535137 8989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:34:07.539757 8989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:34:07.539857 8989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:34:07.539955 8989 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.53.193
I20260926 07:34:07.545482 8989 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:45565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407783529717-22743-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407783529717-22743-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.53.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407783529717-22743-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.53.193
--webserver_port=0
--tserver_master_addrs=127.22.53.254:36349
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8989
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.53.193
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407783529717-22743-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:34:07.546725 8989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:34:07.548234 8989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:34:07.559258 8997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:07.559266 8994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:07.559266 8995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:34:07.559809 8989 server_base.cc:1034] running on GCE node
I20260926 07:34:07.560523 8989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:34:07.561545 8989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:34:07.562783 8989 hybrid_clock.cc:648] HybridClock initialized: now 1790408047562759 us; error 42 us; skew 500 ppm
I20260926 07:34:07.570588 8989 webserver.cc:460] Webserver started at http://127.22.53.193:34827/ using document root <none> and password file <none>
I20260926 07:34:07.571296 8989 fs_manager.cc:359] Metadata directory not provided
I20260926 07:34:07.571372 8989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:34:07.571604 8989 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:34:07.573990 8989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407783529717-22743-0/minicluster-data/ts-0/data/instance:
uuid: "fa449c1fb10a422e974e0536d0be1380"
format_stamp: "Formatted at 2026-09-26 07:34:07 on dist-test-slave-wdjc"
server_key: "bfca90045b1cded4f2f37ffa3cbf195f"
server_key_iv: "eed9b715e5922a7df47b2f9139fc9434"
server_key_version: "encryptionkey@0"
I20260926 07:34:07.574625 8989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407783529717-22743-0/minicluster-data/ts-0/wal/instance:
uuid: "fa449c1fb10a422e974e0536d0be1380"
format_stamp: "Formatted at 2026-09-26 07:34:07 on dist-test-slave-wdjc"
server_key: "bfca90045b1cded4f2f37ffa3cbf195f"
server_key_iv: "eed9b715e5922a7df47b2f9139fc9434"
server_key_version: "encryptionkey@0"
I20260926 07:34:07.579932 8989 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260926 07:34:07.583000 9003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:07.584730 8989 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 07:34:07.584909 8989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407783529717-22743-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407783529717-22743-0/minicluster-data/ts-0/wal
uuid: "fa449c1fb10a422e974e0536d0be1380"
format_stamp: "Formatted at 2026-09-26 07:34:07 on dist-test-slave-wdjc"
server_key: "bfca90045b1cded4f2f37ffa3cbf195f"
server_key_iv: "eed9b715e5922a7df47b2f9139fc9434"
server_key_version: "encryptionkey@0"
I20260926 07:34:07.585059 8989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407783529717-22743-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407783529717-22743-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407783529717-22743-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:34:07.606343 8989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:34:07.607116 8989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:34:07.607335 8989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:34:07.608027 8989 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:34:07.609129 8989 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:34:07.609203 8989 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:07.609301 8989 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:34:07.609346 8989 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:07.620440 8989 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.193:34965
I20260926 07:34:07.620467 9116 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.193:34965 every 8 connection(s)
I20260926 07:34:07.621562 8989 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407783529717-22743-0/minicluster-data/ts-0/data/info.pb
I20260926 07:34:07.631201 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 8989
I20260926 07:34:07.631374 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407783529717-22743-0/minicluster-data/ts-0/wal/instance
I20260926 07:34:07.631734 22743 external_mini_cluster.cc:1428] Setting key 95e0ba2e7136f4fed8d955d016953375
I20260926 07:34:07.632390 8968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:07.623606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.193:53557 (local address 127.22.53.254:36349)
0926 07:34:07.623787 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:07.623792 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:07.625175 (+ 1383us) server_negotiation.cc:407] Connection header received
0926 07:34:07.626722 (+ 1547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:07.626726 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:07.626796 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:07.626879 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:07.627911 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:07.628496 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:07.629826 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:07.630072 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:07.630213 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:07.631783 (+ 1570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:07.631788 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:07.631791 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:34:07.631814 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:07.631841 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:34:07.631853 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:07.631906 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:07.632170 (+ 264us) server_negotiation.cc:299] Negotiation successful
0926 07:34:07.632261 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 07:34:07.633950 9119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:07.624074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:36349 (local address 127.22.53.193:53557)
0926 07:34:07.624970 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:07.625031 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:07.626450 (+ 1419us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:07.627076 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:07.627084 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:07.627182 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:07.627742 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:07.627755 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:07.628706 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:07.628710 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:07.629667 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:07.629677 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:07.630268 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:07.630273 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:07.630464 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:07.631375 (+ 911us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:34:07.631378 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:07.631442 (+ 64us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:34:07.631548 (+ 106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:34:07.631982 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:34:07.631992 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:34:07.631997 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 07:34:07.632620 (+ 623us) client_negotiation.cc:240] Negotiation successful
0926 07:34:07.632952 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":156,"threads_started":1}
I20260926 07:34:07.634547 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407783529717-22743-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407783529717-22743-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407783529717-22743-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407783529717-22743-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.53.194:0
--local_ip_for_outbound_sockets=127.22.53.194
--webserver_interface=127.22.53.194
--webserver_port=0
--tserver_master_addrs=127.22.53.254:36349
--builtin_ntp_servers=127.22.53.212:45565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 07:34:07.635757 9117 heartbeater.cc:347] Connected to a master server at 127.22.53.254:36349
I20260926 07:34:07.636066 9117 heartbeater.cc:464] Registering TS with master...
I20260926 07:34:07.636798 9117 heartbeater.cc:511] Master 127.22.53.254:36349 requested a full tablet report, sending...
I20260926 07:34:07.638944 8928 ts_manager.cc:194] Registered new tserver with Master: fa449c1fb10a422e974e0536d0be1380 (127.22.53.193:34965)
I20260926 07:34:07.641181 8928 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.53.193:53557
W20260926 07:34:07.753109 9120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:34:07.753448 9120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:34:07.753522 9120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:34:07.758245 9120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:34:07.758350 9120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:34:07.758445 9120 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.53.194
I20260926 07:34:07.763712 9120 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:45565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407783529717-22743-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407783529717-22743-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.53.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407783529717-22743-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.53.194
--webserver_port=0
--tserver_master_addrs=127.22.53.254:36349
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9120
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.53.194
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407783529717-22743-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:34:07.765007 9120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:34:07.766487 9120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:34:07.776678 9125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:34:07.776793 9120 server_base.cc:1034] running on GCE node
W20260926 07:34:07.776678 9126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:07.776678 9128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:34:07.777498 9120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:34:07.778231 9120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:34:07.779433 9120 hybrid_clock.cc:648] HybridClock initialized: now 1790408047779417 us; error 31 us; skew 500 ppm
I20260926 07:34:07.781486 9120 webserver.cc:460] Webserver started at http://127.22.53.194:39489/ using document root <none> and password file <none>
I20260926 07:34:07.782145 9120 fs_manager.cc:359] Metadata directory not provided
I20260926 07:34:07.782244 9120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:34:07.782488 9120 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:34:07.784718 9120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407783529717-22743-0/minicluster-data/ts-1/data/instance:
uuid: "96337fc0f92b4d9ca777642a015fead4"
format_stamp: "Formatted at 2026-09-26 07:34:07 on dist-test-slave-wdjc"
server_key: "9af365b991757612758386e48572811b"
server_key_iv: "1356f3d9f9bbf2f500c23ddd65582bd5"
server_key_version: "encryptionkey@0"
I20260926 07:34:07.785339 9120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407783529717-22743-0/minicluster-data/ts-1/wal/instance:
uuid: "96337fc0f92b4d9ca777642a015fead4"
format_stamp: "Formatted at 2026-09-26 07:34:07 on dist-test-slave-wdjc"
server_key: "9af365b991757612758386e48572811b"
server_key_iv: "1356f3d9f9bbf2f500c23ddd65582bd5"
server_key_version: "encryptionkey@0"
I20260926 07:34:07.789553 9120 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 07:34:07.792464 9134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:07.793903 9120 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:34:07.794077 9120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407783529717-22743-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407783529717-22743-0/minicluster-data/ts-1/wal
uuid: "96337fc0f92b4d9ca777642a015fead4"
format_stamp: "Formatted at 2026-09-26 07:34:07 on dist-test-slave-wdjc"
server_key: "9af365b991757612758386e48572811b"
server_key_iv: "1356f3d9f9bbf2f500c23ddd65582bd5"
server_key_version: "encryptionkey@0"
I20260926 07:34:07.794217 9120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407783529717-22743-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407783529717-22743-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407783529717-22743-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:34:07.810654 9120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:34:07.811501 9120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:34:07.811720 9120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:34:07.812404 9120 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:34:07.813537 9120 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:34:07.813609 9120 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:07.813683 9120 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:34:07.813724 9120 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:07.822877 9120 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.194:33311
I20260926 07:34:07.822916 9247 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.194:33311 every 8 connection(s)
I20260926 07:34:07.823848 9120 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407783529717-22743-0/minicluster-data/ts-1/data/info.pb
I20260926 07:34:07.832803 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 9120
I20260926 07:34:07.832955 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407783529717-22743-0/minicluster-data/ts-1/wal/instance
I20260926 07:34:07.833325 22743 external_mini_cluster.cc:1428] Setting key b0d94f93bb5f5c385fa9acceaf58ab31
I20260926 07:34:07.834508 8968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:07.826142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.194:60039 (local address 127.22.53.254:36349)
0926 07:34:07.826346 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:07.826350 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:07.827389 (+ 1039us) server_negotiation.cc:407] Connection header received
0926 07:34:07.828885 (+ 1496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:07.828888 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:07.828952 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:07.829042 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:07.830063 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:07.830655 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:07.831917 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:07.832141 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:07.832239 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:07.833884 (+ 1645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:07.833922 (+ 38us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:07.833928 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:34:07.833946 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:07.833974 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:34:07.833985 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:07.834075 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:07.834325 (+ 250us) server_negotiation.cc:299] Negotiation successful
0926 07:34:07.834389 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260926 07:34:07.835618 9250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:07.826536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:36349 (local address 127.22.53.194:60039)
0926 07:34:07.827248 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:07.827293 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:07.828635 (+ 1342us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:07.829278 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:07.829286 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:07.829376 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:07.829902 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:07.829915 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:07.830819 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:07.830822 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:07.831777 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:07.831788 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:07.832322 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:07.832326 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:07.832512 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:07.833495 (+ 983us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:34:07.833497 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:07.833545 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:34:07.833622 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:34:07.834090 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:34:07.834120 (+ 30us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:34:07.834129 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 07:34:07.834280 (+ 151us) client_negotiation.cc:240] Negotiation successful
0926 07:34:07.834617 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":125,"threads_started":1}
I20260926 07:34:07.835633 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407783529717-22743-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407783529717-22743-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407783529717-22743-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407783529717-22743-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.53.195:0
--local_ip_for_outbound_sockets=127.22.53.195
--webserver_interface=127.22.53.195
--webserver_port=0
--tserver_master_addrs=127.22.53.254:36349
--builtin_ntp_servers=127.22.53.212:45565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 07:34:07.836990 9248 heartbeater.cc:347] Connected to a master server at 127.22.53.254:36349
I20260926 07:34:07.837303 9248 heartbeater.cc:464] Registering TS with master...
I20260926 07:34:07.837908 9248 heartbeater.cc:511] Master 127.22.53.254:36349 requested a full tablet report, sending...
I20260926 07:34:07.839176 8928 ts_manager.cc:194] Registered new tserver with Master: 96337fc0f92b4d9ca777642a015fead4 (127.22.53.194:33311)
I20260926 07:34:07.839859 8928 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.53.194:60039
W20260926 07:34:07.953881 9251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:34:07.954176 9251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:34:07.954218 9251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:34:07.958726 9251 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:34:07.958812 9251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:34:07.958873 9251 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.53.195
I20260926 07:34:07.964432 9251 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:45565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407783529717-22743-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407783529717-22743-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.53.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407783529717-22743-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.53.195
--webserver_port=0
--tserver_master_addrs=127.22.53.254:36349
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9251
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.53.195
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407783529717-22743-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:34:07.965674 9251 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:34:07.967083 9251 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:34:07.977646 9257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:07.977903 9259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:07.977931 9256 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:34:07.978047 9251 server_base.cc:1034] running on GCE node
I20260926 07:34:07.978533 9251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:34:07.979251 9251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:34:07.980461 9251 hybrid_clock.cc:648] HybridClock initialized: now 1790408047980444 us; error 33 us; skew 500 ppm
I20260926 07:34:07.982576 9251 webserver.cc:460] Webserver started at http://127.22.53.195:45749/ using document root <none> and password file <none>
I20260926 07:34:07.983233 9251 fs_manager.cc:359] Metadata directory not provided
I20260926 07:34:07.983338 9251 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:34:07.983585 9251 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:34:07.986054 9251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407783529717-22743-0/minicluster-data/ts-2/data/instance:
uuid: "458277a2eed8412aaf81a0b8a5060255"
format_stamp: "Formatted at 2026-09-26 07:34:07 on dist-test-slave-wdjc"
server_key: "13bb644902e03ed08b129f4b4655664e"
server_key_iv: "dc106fd5f15f805b827ddfb707793638"
server_key_version: "encryptionkey@0"
I20260926 07:34:07.986696 9251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407783529717-22743-0/minicluster-data/ts-2/wal/instance:
uuid: "458277a2eed8412aaf81a0b8a5060255"
format_stamp: "Formatted at 2026-09-26 07:34:07 on dist-test-slave-wdjc"
server_key: "13bb644902e03ed08b129f4b4655664e"
server_key_iv: "dc106fd5f15f805b827ddfb707793638"
server_key_version: "encryptionkey@0"
I20260926 07:34:07.991308 9251 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 07:34:07.994269 9265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:07.995752 9251 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 07:34:07.995934 9251 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407783529717-22743-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407783529717-22743-0/minicluster-data/ts-2/wal
uuid: "458277a2eed8412aaf81a0b8a5060255"
format_stamp: "Formatted at 2026-09-26 07:34:07 on dist-test-slave-wdjc"
server_key: "13bb644902e03ed08b129f4b4655664e"
server_key_iv: "dc106fd5f15f805b827ddfb707793638"
server_key_version: "encryptionkey@0"
I20260926 07:34:07.996081 9251 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407783529717-22743-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407783529717-22743-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407783529717-22743-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:34:08.032644 9251 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:34:08.033632 9251 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:34:08.033898 9251 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:34:08.034560 9251 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:34:08.035622 9251 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:34:08.035701 9251 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:08.035775 9251 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:34:08.035827 9251 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:08.045439 9251 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.195:37501
I20260926 07:34:08.045506 9378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.195:37501 every 8 connection(s)
I20260926 07:34:08.046452 9251 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407783529717-22743-0/minicluster-data/ts-2/data/info.pb
I20260926 07:34:08.054091 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 9251
I20260926 07:34:08.054256 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407783529717-22743-0/minicluster-data/ts-2/wal/instance
I20260926 07:34:08.054574 22743 external_mini_cluster.cc:1428] Setting key 39914e6328ca14faa138b5616c7f4c64
I20260926 07:34:08.056659 8968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:08.048429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.195:44179 (local address 127.22.53.254:36349)
0926 07:34:08.048626 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:08.048631 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:08.049656 (+ 1025us) server_negotiation.cc:407] Connection header received
0926 07:34:08.051123 (+ 1467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:08.051127 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:08.051191 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:08.051278 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:08.052259 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:08.052829 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:08.054272 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:08.054532 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:08.054631 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:08.056086 (+ 1455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:08.056094 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:08.056097 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:34:08.056109 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:08.056129 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:34:08.056138 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:08.056220 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:08.056404 (+ 184us) server_negotiation.cc:299] Negotiation successful
0926 07:34:08.056475 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 07:34:08.057649 9381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:08.048803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:36349 (local address 127.22.53.195:44179)
0926 07:34:08.049482 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:08.049525 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:08.050885 (+ 1360us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:08.051468 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:08.051476 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:08.051565 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:08.052082 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:08.052096 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:08.052994 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:08.052997 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:08.054073 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:08.054086 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:08.054720 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:08.054724 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:08.054966 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:08.055745 (+ 779us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:34:08.055747 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:08.055817 (+ 70us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:34:08.055918 (+ 101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:34:08.056223 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:34:08.056235 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:34:08.056243 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 07:34:08.056394 (+ 151us) client_negotiation.cc:240] Negotiation successful
0926 07:34:08.056750 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":114,"threads_started":1}
I20260926 07:34:08.058866 9379 heartbeater.cc:347] Connected to a master server at 127.22.53.254:36349
I20260926 07:34:08.059144 9379 heartbeater.cc:464] Registering TS with master...
I20260926 07:34:08.059747 9379 heartbeater.cc:511] Master 127.22.53.254:36349 requested a full tablet report, sending...
I20260926 07:34:08.060943 8928 ts_manager.cc:194] Registered new tserver with Master: 458277a2eed8412aaf81a0b8a5060255 (127.22.53.195:37501)
I20260926 07:34:08.061565 8928 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.53.195:44179
I20260926 07:34:08.069504 22743 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:34:08.075824 8968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:08.071646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50326 (local address 127.22.53.254:36349)
0926 07:34:08.071780 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:08.071786 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:08.071967 (+ 181us) server_negotiation.cc:407] Connection header received
0926 07:34:08.072130 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:08.072138 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:08.072182 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:08.072272 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:08.072591 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:08.073111 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:08.073994 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:08.074216 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:08.074336 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:08.075275 (+ 939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:08.075279 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:08.075281 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:34:08.075294 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:08.075326 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:34:08.075335 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:08.075482 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:08.075622 (+ 140us) server_negotiation.cc:299] Negotiation successful
0926 07:34:08.075665 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 07:34:08.079121 8928 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50326:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:34:08.082470 8928 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:34:08.099253 9390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:08.092704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.194:33311 (local address 127.0.0.1:53684)
0926 07:34:08.093079 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:08.093140 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:08.093284 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:08.094169 (+ 885us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:08.094176 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:08.094200 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:34:08.094394 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:08.094403 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:08.095608 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:08.095612 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:08.097135 (+ 1523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:08.097149 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:08.098574 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:08.098578 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:08.098699 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:08.098771 (+ 72us) client_negotiation.cc:769] Sending connection context
0926 07:34:08.098885 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 07:34:08.099087 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"mutex_wait_us":55,"thread_start_us":99,"threads_started":1}
I20260926 07:34:08.099267 9391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:08.092842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53684 (local address 127.22.53.194:33311)
0926 07:34:08.093317 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:08.093330 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:08.093357 (+ 27us) server_negotiation.cc:407] Connection header received
0926 07:34:08.093438 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:08.093443 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:08.093678 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:08.093949 (+ 271us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:34:08.094542 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:08.095487 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:08.097286 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:08.098183 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:08.098296 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:08.098513 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:08.098951 (+ 438us) server_negotiation.cc:299] Negotiation successful
0926 07:34:08.099079 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":129,"threads_started":1}
I20260926 07:34:08.099507 9395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:08.094444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54274 (local address 127.22.53.193:34965)
0926 07:34:08.094700 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:08.094707 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:08.094728 (+ 21us) server_negotiation.cc:407] Connection header received
0926 07:34:08.094805 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:08.094809 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:08.094972 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:08.095136 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:34:08.095385 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:08.096376 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:08.097728 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:08.098453 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:08.098581 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:08.098773 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:08.099074 (+ 301us) server_negotiation.cc:299] Negotiation successful
0926 07:34:08.099231 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":93,"threads_started":1}
I20260926 07:34:08.099916 9393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:08.093817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.193:34965 (local address 127.0.0.1:54274)
0926 07:34:08.094021 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:08.094033 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:08.094108 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:08.095112 (+ 1004us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:08.095115 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:08.095134 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:34:08.095234 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:08.095240 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:08.096497 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:08.096503 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:08.097572 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:08.097584 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:08.098594 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:08.098598 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:08.098724 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:08.098824 (+ 100us) client_negotiation.cc:769] Sending connection context
0926 07:34:08.098915 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 07:34:08.099769 (+ 854us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":619,"thread_start_us":96,"threads_started":1}
I20260926 07:34:08.100621 9392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:08.093465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.195:37501 (local address 127.0.0.1:51950)
0926 07:34:08.093718 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:08.093743 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:08.093833 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:08.095910 (+ 2077us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:08.095913 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:08.095934 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:34:08.096028 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:08.096034 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:08.097491 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:08.097494 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:08.098627 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:08.098636 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:08.100214 (+ 1578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:08.100217 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:08.100322 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:08.100348 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 07:34:08.100400 (+ 52us) client_negotiation.cc:240] Negotiation successful
0926 07:34:08.100470 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":121,"threads_started":1}
I20260926 07:34:08.101579 9394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:08.093645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51950 (local address 127.22.53.195:37501)
0926 07:34:08.095455 (+ 1810us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:08.095464 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:08.095483 (+ 19us) server_negotiation.cc:407] Connection header received
0926 07:34:08.095542 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:08.095546 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:08.095749 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:08.095899 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:34:08.096160 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:08.097371 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:08.098787 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:08.100077 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:08.101052 (+ 975us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:08.101268 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:08.101321 (+ 53us) server_negotiation.cc:299] Negotiation successful
0926 07:34:08.101421 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1709,"thread_start_us":299,"threads_started":1}
I20260926 07:34:08.102782 9051 tablet_service.cc:1467] Processing CreateTablet for tablet 189f06d57cda43df902347f308c4b9a4 (DEFAULT_TABLE table=test-table [id=97bee0b9668e4f7d8d44b3d4ec68c272]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:34:08.102782 9182 tablet_service.cc:1467] Processing CreateTablet for tablet 189f06d57cda43df902347f308c4b9a4 (DEFAULT_TABLE table=test-table [id=97bee0b9668e4f7d8d44b3d4ec68c272]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:34:08.103838 9051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 189f06d57cda43df902347f308c4b9a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:34:08.103850 9182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 189f06d57cda43df902347f308c4b9a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:34:08.104478 9313 tablet_service.cc:1467] Processing CreateTablet for tablet 189f06d57cda43df902347f308c4b9a4 (DEFAULT_TABLE table=test-table [id=97bee0b9668e4f7d8d44b3d4ec68c272]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:34:08.105813 9313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 189f06d57cda43df902347f308c4b9a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:34:08.112361 9397 tablet_bootstrap.cc:492] T 189f06d57cda43df902347f308c4b9a4 P 96337fc0f92b4d9ca777642a015fead4: Bootstrap starting.
I20260926 07:34:08.112361 9396 tablet_bootstrap.cc:492] T 189f06d57cda43df902347f308c4b9a4 P fa449c1fb10a422e974e0536d0be1380: Bootstrap starting.
I20260926 07:34:08.114240 9398 tablet_bootstrap.cc:492] T 189f06d57cda43df902347f308c4b9a4 P 458277a2eed8412aaf81a0b8a5060255: Bootstrap starting.
I20260926 07:34:08.114962 9397 tablet_bootstrap.cc:654] T 189f06d57cda43df902347f308c4b9a4 P 96337fc0f92b4d9ca777642a015fead4: Neither blocks nor log segments found. Creating new log.
I20260926 07:34:08.114962 9396 tablet_bootstrap.cc:654] T 189f06d57cda43df902347f308c4b9a4 P fa449c1fb10a422e974e0536d0be1380: Neither blocks nor log segments found. Creating new log.
I20260926 07:34:08.115936 9396 log.cc:824] T 189f06d57cda43df902347f308c4b9a4 P fa449c1fb10a422e974e0536d0be1380: Log is configured to *not* fsync() on all Append() calls
I20260926 07:34:08.115937 9397 log.cc:824] T 189f06d57cda43df902347f308c4b9a4 P 96337fc0f92b4d9ca777642a015fead4: Log is configured to *not* fsync() on all Append() calls
I20260926 07:34:08.116456 9398 tablet_bootstrap.cc:654] T 189f06d57cda43df902347f308c4b9a4 P 458277a2eed8412aaf81a0b8a5060255: Neither blocks nor log segments found. Creating new log.
I20260926 07:34:08.117385 9398 log.cc:824] T 189f06d57cda43df902347f308c4b9a4 P 458277a2eed8412aaf81a0b8a5060255: Log is configured to *not* fsync() on all Append() calls
I20260926 07:34:08.118335 9396 tablet_bootstrap.cc:492] T 189f06d57cda43df902347f308c4b9a4 P fa449c1fb10a422e974e0536d0be1380: No bootstrap required, opened a new log
I20260926 07:34:08.118389 9397 tablet_bootstrap.cc:492] T 189f06d57cda43df902347f308c4b9a4 P 96337fc0f92b4d9ca777642a015fead4: No bootstrap required, opened a new log
I20260926 07:34:08.118569 9396 ts_tablet_manager.cc:1397] T 189f06d57cda43df902347f308c4b9a4 P fa449c1fb10a422e974e0536d0be1380: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 07:34:08.118573 9397 ts_tablet_manager.cc:1397] T 189f06d57cda43df902347f308c4b9a4 P 96337fc0f92b4d9ca777642a015fead4: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 07:34:08.119477 9398 tablet_bootstrap.cc:492] T 189f06d57cda43df902347f308c4b9a4 P 458277a2eed8412aaf81a0b8a5060255: No bootstrap required, opened a new log
I20260926 07:34:08.119675 9398 ts_tablet_manager.cc:1397] T 189f06d57cda43df902347f308c4b9a4 P 458277a2eed8412aaf81a0b8a5060255: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 07:34:08.122205 9396 raft_consensus.cc:348] T 189f06d57cda43df902347f308c4b9a4 P fa449c1fb10a422e974e0536d0be1380 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96337fc0f92b4d9ca777642a015fead4" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 33311 } } peers { permanent_uuid: "458277a2eed8412aaf81a0b8a5060255" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 37501 } } peers { permanent_uuid: "fa449c1fb10a422e974e0536d0be1380" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 34965 } }
I20260926 07:34:08.122510 9396 raft_consensus.cc:374] T 189f06d57cda43df902347f308c4b9a4 P fa449c1fb10a422e974e0536d0be1380 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:34:08.122596 9396 raft_consensus.cc:729] T 189f06d57cda43df902347f308c4b9a4 P fa449c1fb10a422e974e0536d0be1380 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa449c1fb10a422e974e0536d0be1380, State: Initialized, Role: FOLLOWER
I20260926 07:34:08.123116 9396 consensus_queue.cc:260] T 189f06d57cda43df902347f308c4b9a4 P fa449c1fb10a422e974e0536d0be1380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96337fc0f92b4d9ca777642a015fead4" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 33311 } } peers { permanent_uuid: "458277a2eed8412aaf81a0b8a5060255" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 37501 } } peers { permanent_uuid: "fa449c1fb10a422e974e0536d0be1380" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 34965 } }
I20260926 07:34:08.123225 9397 raft_consensus.cc:348] T 189f06d57cda43df902347f308c4b9a4 P 96337fc0f92b4d9ca777642a015fead4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96337fc0f92b4d9ca777642a015fead4" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 33311 } } peers { permanent_uuid: "458277a2eed8412aaf81a0b8a5060255" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 37501 } } peers { permanent_uuid: "fa449c1fb10a422e974e0536d0be1380" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 34965 } }
I20260926 07:34:08.123579 9397 raft_consensus.cc:374] T 189f06d57cda43df902347f308c4b9a4 P 96337fc0f92b4d9ca777642a015fead4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:34:08.123675 9397 raft_consensus.cc:729] T 189f06d57cda43df902347f308c4b9a4 P 96337fc0f92b4d9ca777642a015fead4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96337fc0f92b4d9ca777642a015fead4, State: Initialized, Role: FOLLOWER
I20260926 07:34:08.123982 9398 raft_consensus.cc:348] T 189f06d57cda43df902347f308c4b9a4 P 458277a2eed8412aaf81a0b8a5060255 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96337fc0f92b4d9ca777642a015fead4" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 33311 } } peers { permanent_uuid: "458277a2eed8412aaf81a0b8a5060255" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 37501 } } peers { permanent_uuid: "fa449c1fb10a422e974e0536d0be1380" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 34965 } }
I20260926 07:34:08.124306 9396 ts_tablet_manager.cc:1428] T 189f06d57cda43df902347f308c4b9a4 P fa449c1fb10a422e974e0536d0be1380: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260926 07:34:08.124213 9397 consensus_queue.cc:260] T 189f06d57cda43df902347f308c4b9a4 P 96337fc0f92b4d9ca777642a015fead4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96337fc0f92b4d9ca777642a015fead4" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 33311 } } peers { permanent_uuid: "458277a2eed8412aaf81a0b8a5060255" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 37501 } } peers { permanent_uuid: "fa449c1fb10a422e974e0536d0be1380" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 34965 } }
I20260926 07:34:08.124286 9398 raft_consensus.cc:374] T 189f06d57cda43df902347f308c4b9a4 P 458277a2eed8412aaf81a0b8a5060255 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:34:08.124455 9398 raft_consensus.cc:729] T 189f06d57cda43df902347f308c4b9a4 P 458277a2eed8412aaf81a0b8a5060255 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 458277a2eed8412aaf81a0b8a5060255, State: Initialized, Role: FOLLOWER
I20260926 07:34:08.124780 9117 heartbeater.cc:503] Master 127.22.53.254:36349 was elected leader, sending a full tablet report...
I20260926 07:34:08.125061 9398 consensus_queue.cc:260] T 189f06d57cda43df902347f308c4b9a4 P 458277a2eed8412aaf81a0b8a5060255 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96337fc0f92b4d9ca777642a015fead4" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 33311 } } peers { permanent_uuid: "458277a2eed8412aaf81a0b8a5060255" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 37501 } } peers { permanent_uuid: "fa449c1fb10a422e974e0536d0be1380" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 34965 } }
I20260926 07:34:08.125504 9397 ts_tablet_manager.cc:1428] T 189f06d57cda43df902347f308c4b9a4 P 96337fc0f92b4d9ca777642a015fead4: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260926 07:34:08.125937 9379 heartbeater.cc:503] Master 127.22.53.254:36349 was elected leader, sending a full tablet report...
I20260926 07:34:08.126037 9248 heartbeater.cc:503] Master 127.22.53.254:36349 was elected leader, sending a full tablet report...
I20260926 07:34:08.126461 9398 ts_tablet_manager.cc:1428] T 189f06d57cda43df902347f308c4b9a4 P 458277a2eed8412aaf81a0b8a5060255: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260926 07:34:08.285863 9403 raft_consensus.cc:482] T 189f06d57cda43df902347f308c4b9a4 P 96337fc0f92b4d9ca777642a015fead4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:34:08.286082 9403 raft_consensus.cc:504] T 189f06d57cda43df902347f308c4b9a4 P 96337fc0f92b4d9ca777642a015fead4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96337fc0f92b4d9ca777642a015fead4" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 33311 } } peers { permanent_uuid: "458277a2eed8412aaf81a0b8a5060255" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 37501 } } peers { permanent_uuid: "fa449c1fb10a422e974e0536d0be1380" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 34965 } }
I20260926 07:34:08.287277 9403 leader_election.cc:288] T 189f06d57cda43df902347f308c4b9a4 P 96337fc0f92b4d9ca777642a015fead4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 458277a2eed8412aaf81a0b8a5060255 (127.22.53.195:37501), fa449c1fb10a422e974e0536d0be1380 (127.22.53.193:34965)
I20260926 07:34:08.291499 9250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:08.287587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.195:37501 (local address 127.22.53.194:47649)
0926 07:34:08.287752 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:08.287766 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:08.287894 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:08.288186 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:08.288189 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:08.288207 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:34:08.288324 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:08.288330 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:08.289191 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:08.289218 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:08.290250 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:08.290261 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:08.291056 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:08.291059 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:08.291163 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:08.291192 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 07:34:08.291280 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 07:34:08.291358 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
I20260926 07:34:08.291499 9394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:08.287745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.194:47649 (local address 127.22.53.195:37501)
0926 07:34:08.287891 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:08.287896 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:08.287912 (+ 16us) server_negotiation.cc:407] Connection header received
0926 07:34:08.287992 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:08.287998 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:08.288043 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:08.288143 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:34:08.288443 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:08.289091 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:08.290426 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:08.290929 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:08.291055 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:08.291110 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:08.291298 (+ 188us) server_negotiation.cc:299] Negotiation successful
0926 07:34:08.291372 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 07:34:08.292829 9395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:08.288072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.194:33405 (local address 127.22.53.193:34965)
0926 07:34:08.288253 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:08.288259 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:08.288760 (+ 501us) server_negotiation.cc:407] Connection header received
0926 07:34:08.289208 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:08.289212 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:08.289287 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:08.289385 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:34:08.289708 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:08.290426 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:08.291607 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:08.292182 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:08.292334 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:08.292387 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:08.292650 (+ 263us) server_negotiation.cc:299] Negotiation successful
0926 07:34:08.292728 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 07:34:08.292891 9405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:08.287951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.193:34965 (local address 127.22.53.194:33405)
0926 07:34:08.288403 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:08.288414 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:08.288872 (+ 458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:08.289409 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:08.289412 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:08.289432 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:34:08.289583 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:08.289588 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:08.290530 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:08.290537 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:08.291464 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:08.291473 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:08.292334 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:08.292338 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:08.292450 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:08.292476 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 07:34:08.292615 (+ 139us) client_negotiation.cc:240] Negotiation successful
0926 07:34:08.292701 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":125,"threads_started":1}
I20260926 07:34:08.292824 9333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "189f06d57cda43df902347f308c4b9a4" candidate_uuid: "96337fc0f92b4d9ca777642a015fead4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "458277a2eed8412aaf81a0b8a5060255" is_pre_election: true
I20260926 07:34:08.293221 9333 raft_consensus.cc:2454] T 189f06d57cda43df902347f308c4b9a4 P 458277a2eed8412aaf81a0b8a5060255 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96337fc0f92b4d9ca777642a015fead4 in term 0.
I20260926 07:34:08.293643 9071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "189f06d57cda43df902347f308c4b9a4" candidate_uuid: "96337fc0f92b4d9ca777642a015fead4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa449c1fb10a422e974e0536d0be1380" is_pre_election: true
I20260926 07:34:08.293795 9135 leader_election.cc:302] T 189f06d57cda43df902347f308c4b9a4 P 96337fc0f92b4d9ca777642a015fead4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 458277a2eed8412aaf81a0b8a5060255, 96337fc0f92b4d9ca777642a015fead4; no voters:
I20260926 07:34:08.293998 9071 raft_consensus.cc:2454] T 189f06d57cda43df902347f308c4b9a4 P fa449c1fb10a422e974e0536d0be1380 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96337fc0f92b4d9ca777642a015fead4 in term 0.
I20260926 07:34:08.294088 9403 raft_consensus.cc:2781] T 189f06d57cda43df902347f308c4b9a4 P 96337fc0f92b4d9ca777642a015fead4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:34:08.294189 9403 raft_consensus.cc:482] T 189f06d57cda43df902347f308c4b9a4 P 96337fc0f92b4d9ca777642a015fead4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:34:08.294227 9403 raft_consensus.cc:3037] T 189f06d57cda43df902347f308c4b9a4 P 96337fc0f92b4d9ca777642a015fead4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:34:08.295396 9403 raft_consensus.cc:504] T 189f06d57cda43df902347f308c4b9a4 P 96337fc0f92b4d9ca777642a015fead4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96337fc0f92b4d9ca777642a015fead4" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 33311 } } peers { permanent_uuid: "458277a2eed8412aaf81a0b8a5060255" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 37501 } } peers { permanent_uuid: "fa449c1fb10a422e974e0536d0be1380" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 34965 } }
I20260926 07:34:08.295750 9403 leader_election.cc:288] T 189f06d57cda43df902347f308c4b9a4 P 96337fc0f92b4d9ca777642a015fead4 [CANDIDATE]: Term 1 election: Requested vote from peers 458277a2eed8412aaf81a0b8a5060255 (127.22.53.195:37501), fa449c1fb10a422e974e0536d0be1380 (127.22.53.193:34965)
I20260926 07:34:08.296255 9333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "189f06d57cda43df902347f308c4b9a4" candidate_uuid: "96337fc0f92b4d9ca777642a015fead4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "458277a2eed8412aaf81a0b8a5060255"
I20260926 07:34:08.296284 9071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "189f06d57cda43df902347f308c4b9a4" candidate_uuid: "96337fc0f92b4d9ca777642a015fead4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa449c1fb10a422e974e0536d0be1380"
I20260926 07:34:08.296413 9333 raft_consensus.cc:3037] T 189f06d57cda43df902347f308c4b9a4 P 458277a2eed8412aaf81a0b8a5060255 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:34:08.296422 9071 raft_consensus.cc:3037] T 189f06d57cda43df902347f308c4b9a4 P fa449c1fb10a422e974e0536d0be1380 [term 0 FOLLOWER]: Advancing to term 1
W20260926 07:34:08.297464 9380 tablet.cc:2369] T 189f06d57cda43df902347f308c4b9a4 P 458277a2eed8412aaf81a0b8a5060255: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:34:08.297770 9071 raft_consensus.cc:2454] T 189f06d57cda43df902347f308c4b9a4 P fa449c1fb10a422e974e0536d0be1380 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96337fc0f92b4d9ca777642a015fead4 in term 1.
I20260926 07:34:08.298087 9333 raft_consensus.cc:2454] T 189f06d57cda43df902347f308c4b9a4 P 458277a2eed8412aaf81a0b8a5060255 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96337fc0f92b4d9ca777642a015fead4 in term 1.
I20260926 07:34:08.298233 9135 leader_election.cc:302] T 189f06d57cda43df902347f308c4b9a4 P 96337fc0f92b4d9ca777642a015fead4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96337fc0f92b4d9ca777642a015fead4, fa449c1fb10a422e974e0536d0be1380; no voters:
I20260926 07:34:08.298452 9403 raft_consensus.cc:2781] T 189f06d57cda43df902347f308c4b9a4 P 96337fc0f92b4d9ca777642a015fead4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:34:08.298662 9403 raft_consensus.cc:686] T 189f06d57cda43df902347f308c4b9a4 P 96337fc0f92b4d9ca777642a015fead4 [term 1 LEADER]: Becoming Leader. State: Replica: 96337fc0f92b4d9ca777642a015fead4, State: Running, Role: LEADER
I20260926 07:34:08.299041 9403 consensus_queue.cc:237] T 189f06d57cda43df902347f308c4b9a4 P 96337fc0f92b4d9ca777642a015fead4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96337fc0f92b4d9ca777642a015fead4" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 33311 } } peers { permanent_uuid: "458277a2eed8412aaf81a0b8a5060255" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 37501 } } peers { permanent_uuid: "fa449c1fb10a422e974e0536d0be1380" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 34965 } }
I20260926 07:34:08.302680 8928 catalog_manager.cc:5495] T 189f06d57cda43df902347f308c4b9a4 P 96337fc0f92b4d9ca777642a015fead4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 96337fc0f92b4d9ca777642a015fead4 (127.22.53.194). New cstate: current_term: 1 leader_uuid: "96337fc0f92b4d9ca777642a015fead4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96337fc0f92b4d9ca777642a015fead4" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 33311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "458277a2eed8412aaf81a0b8a5060255" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 37501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa449c1fb10a422e974e0536d0be1380" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 34965 } health_report { overall_health: UNKNOWN } } }
W20260926 07:34:08.325156 9249 tablet.cc:2369] T 189f06d57cda43df902347f308c4b9a4 P 96337fc0f92b4d9ca777642a015fead4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:34:08.341235 9391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:08.337279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53688 (local address 127.22.53.194:33311)
0926 07:34:08.337430 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:08.337433 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:08.337448 (+ 15us) server_negotiation.cc:407] Connection header received
0926 07:34:08.337523 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:08.337526 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:08.337581 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:08.337713 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:34:08.337941 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:08.338457 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:08.339410 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:08.339651 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:08.339744 (+ 93us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:08.340140 (+ 396us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:34:08.340702 (+ 562us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:34:08.340829 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:08.341058 (+ 229us) server_negotiation.cc:299] Negotiation successful
0926 07:34:08.341097 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 07:34:08.347827 9333 raft_consensus.cc:1261] T 189f06d57cda43df902347f308c4b9a4 P 458277a2eed8412aaf81a0b8a5060255 [term 1 FOLLOWER]: Refusing update from remote peer 96337fc0f92b4d9ca777642a015fead4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:34:08.347870 9071 raft_consensus.cc:1261] T 189f06d57cda43df902347f308c4b9a4 P fa449c1fb10a422e974e0536d0be1380 [term 1 FOLLOWER]: Refusing update from remote peer 96337fc0f92b4d9ca777642a015fead4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:34:08.348721 9406 consensus_queue.cc:1035] T 189f06d57cda43df902347f308c4b9a4 P 96337fc0f92b4d9ca777642a015fead4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "458277a2eed8412aaf81a0b8a5060255" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 37501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:34:08.348958 9403 consensus_queue.cc:1035] T 189f06d57cda43df902347f308c4b9a4 P 96337fc0f92b4d9ca777642a015fead4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa449c1fb10a422e974e0536d0be1380" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 34965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:34:08.356810 9409 mvcc.cc:205] Tried to move back new op lower bound from 7333511366024536064 to 7333511365835825152. Current Snapshot: MvccSnapshot[applied={T|T < 7333511366024536064}]
I20260926 07:34:08.358636 9410 mvcc.cc:205] Tried to move back new op lower bound from 7333511366024536064 to 7333511365835825152. Current Snapshot: MvccSnapshot[applied={T|T < 7333511366024536064}]
I20260926 07:34:08.367435 8928 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50326:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:34:08.367614 8928 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50326:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:34:08.370560 8928 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a3cf74af4a654b9998bc15bf8d4d2a74: Sending DeleteTablet for 3 replicas of tablet 189f06d57cda43df902347f308c4b9a4
I20260926 07:34:08.371573 9182 tablet_service.cc:1514] Processing DeleteTablet for tablet 189f06d57cda43df902347f308c4b9a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:34:08 UTC) from {username='slave'} at 127.0.0.1:53684
I20260926 07:34:08.371574 9313 tablet_service.cc:1514] Processing DeleteTablet for tablet 189f06d57cda43df902347f308c4b9a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:34:08 UTC) from {username='slave'} at 127.0.0.1:51950
I20260926 07:34:08.371726 9051 tablet_service.cc:1514] Processing DeleteTablet for tablet 189f06d57cda43df902347f308c4b9a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:34:08 UTC) from {username='slave'} at 127.0.0.1:54274
I20260926 07:34:08.372184 9419 tablet_replica.cc:330] T 189f06d57cda43df902347f308c4b9a4 P 96337fc0f92b4d9ca777642a015fead4: stopping tablet replica
I20260926 07:34:08.372596 9419 raft_consensus.cc:2229] T 189f06d57cda43df902347f308c4b9a4 P 96337fc0f92b4d9ca777642a015fead4 [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:34:08.372669 9420 tablet_replica.cc:330] T 189f06d57cda43df902347f308c4b9a4 P fa449c1fb10a422e974e0536d0be1380: stopping tablet replica
I20260926 07:34:08.373025 9420 raft_consensus.cc:2229] T 189f06d57cda43df902347f308c4b9a4 P fa449c1fb10a422e974e0536d0be1380 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:34:08.373054 9419 raft_consensus.cc:2258] T 189f06d57cda43df902347f308c4b9a4 P 96337fc0f92b4d9ca777642a015fead4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:34:08.373112 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 8989
I20260926 07:34:08.373562 9418 tablet_replica.cc:330] T 189f06d57cda43df902347f308c4b9a4 P 458277a2eed8412aaf81a0b8a5060255: stopping tablet replica
I20260926 07:34:08.373910 9418 raft_consensus.cc:2229] T 189f06d57cda43df902347f308c4b9a4 P 458277a2eed8412aaf81a0b8a5060255 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:34:08.374281 9418 raft_consensus.cc:2258] T 189f06d57cda43df902347f308c4b9a4 P 458277a2eed8412aaf81a0b8a5060255 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:34:08.375656 9419 ts_tablet_manager.cc:1905] T 189f06d57cda43df902347f308c4b9a4 P 96337fc0f92b4d9ca777642a015fead4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:34:08.375799 9418 ts_tablet_manager.cc:1905] T 189f06d57cda43df902347f308c4b9a4 P 458277a2eed8412aaf81a0b8a5060255: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:34:08.378334 9419 ts_tablet_manager.cc:1918] T 189f06d57cda43df902347f308c4b9a4 P 96337fc0f92b4d9ca777642a015fead4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:34:08.378486 9419 log.cc:1196] T 189f06d57cda43df902347f308c4b9a4 P 96337fc0f92b4d9ca777642a015fead4: Deleting WAL directory at /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407783529717-22743-0/minicluster-data/ts-1/wal/wals/189f06d57cda43df902347f308c4b9a4
I20260926 07:34:08.378965 9419 ts_tablet_manager.cc:1939] T 189f06d57cda43df902347f308c4b9a4 P 96337fc0f92b4d9ca777642a015fead4: Deleting consensus metadata
I20260926 07:34:08.379160 9418 ts_tablet_manager.cc:1918] T 189f06d57cda43df902347f308c4b9a4 P 458277a2eed8412aaf81a0b8a5060255: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:34:08.379312 9418 log.cc:1196] T 189f06d57cda43df902347f308c4b9a4 P 458277a2eed8412aaf81a0b8a5060255: Deleting WAL directory at /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407783529717-22743-0/minicluster-data/ts-2/wal/wals/189f06d57cda43df902347f308c4b9a4
I20260926 07:34:08.379643 9418 ts_tablet_manager.cc:1939] T 189f06d57cda43df902347f308c4b9a4 P 458277a2eed8412aaf81a0b8a5060255: Deleting consensus metadata
I20260926 07:34:08.380453 8913 catalog_manager.cc:4841] TS 96337fc0f92b4d9ca777642a015fead4 (127.22.53.194:33311): tablet 189f06d57cda43df902347f308c4b9a4 (table test-table [id=97bee0b9668e4f7d8d44b3d4ec68c272]) successfully deleted
I20260926 07:34:08.380901 8913 catalog_manager.cc:4841] TS 458277a2eed8412aaf81a0b8a5060255 (127.22.53.195:37501): tablet 189f06d57cda43df902347f308c4b9a4 (table test-table [id=97bee0b9668e4f7d8d44b3d4ec68c272]) successfully deleted
W20260926 07:34:08.382576 8913 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.53.193:34965 (error 108)
I20260926 07:34:08.382653 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 9120
I20260926 07:34:08.384159 9393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:08.383760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.193:34965 (local address 127.0.0.1:54286)
0926 07:34:08.383951 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:08.384044 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.193:34965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260926 07:34:08.384358 8913 catalog_manager.cc:4584] TS fa449c1fb10a422e974e0536d0be1380 (127.22.53.193:34965): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 189f06d57cda43df902347f308c4b9a4: Network error: Client connection negotiation failed: client connection to 127.22.53.193:34965: connect: Connection refused (error 111)
I20260926 07:34:08.389350 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 9251
I20260926 07:34:08.395190 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 8898
2026-09-26T07:34:08Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1348 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407783529717-22743-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:34:08 dist-test-slave-wdjc krb5kdc[9424](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:34:08 dist-test-slave-wdjc krb5kdc[9424](info): set up 2 sockets
Sep 26 07:34:08 dist-test-slave-wdjc krb5kdc[9424](info): commencing operation
krb5kdc: starting...
W20260926 07:34:09.488914 22743 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407783529717-22743-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407783529717-22743-0/krb5kdc/test-user.keytab.
Sep 26 07:34:09 dist-test-slave-wdjc krb5kdc[9424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408049, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:34:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:34:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.53.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.53.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.53.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab.
Entry for principal kudu/127.22.53.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab.
I20260926 07:34:09.621862 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407783529717-22743-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407783529717-22743-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407783529717-22743-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.53.254:33575
--webserver_interface=127.22.53.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab
--principal=kudu/127.22.53.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:38951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.53.254:33575
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:34:09.741508 9437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:34:09.741888 9437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:34:09.741989 9437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:34:09.746935 9437 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:34:09.747071 9437 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:34:09.747107 9437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:34:09.747145 9437 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:34:09.747182 9437 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:34:09.753103 9437 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:38951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407783529717-22743-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407783529717-22743-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.53.254:33575
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab
--principal=kudu/127.22.53.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.53.254:33575
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.53.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9437
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407783529717-22743-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:34:09.754366 9437 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:34:09.755527 9437 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:34:09.765272 9445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:09.765260 9442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:09.765260 9443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:34:09.765697 9437 server_base.cc:1034] running on GCE node
I20260926 07:34:09.766378 9437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:34:09.767627 9437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:34:09.768875 9437 hybrid_clock.cc:648] HybridClock initialized: now 1790408049768831 us; error 63 us; skew 500 ppm
Sep 26 07:34:09 dist-test-slave-wdjc krb5kdc[9424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408049, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:34:09.772431 9437 init.cc:377] Logged in from keytab as kudu/127.22.53.254@KRBTEST.COM (short username kudu)
I20260926 07:34:09.773561 9437 webserver.cc:460] Webserver started at http://127.22.53.254:34289/ using document root <none> and password file <none>
I20260926 07:34:09.774168 9437 fs_manager.cc:359] Metadata directory not provided
I20260926 07:34:09.774243 9437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:34:09.774446 9437 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:34:09.777487 9437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407783529717-22743-0/minicluster-data/master-0/data/instance:
uuid: "bd1b7a31c22d43919807b0d3e370e75a"
format_stamp: "Formatted at 2026-09-26 07:34:09 on dist-test-slave-wdjc"
server_key: "b33698c1fe2c4d8127ab0576f4118073"
server_key_iv: "e882ba0141fd1c6c043ff6f05b54a485"
server_key_version: "encryptionkey@0"
I20260926 07:34:09.778143 9437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407783529717-22743-0/minicluster-data/master-0/wal/instance:
uuid: "bd1b7a31c22d43919807b0d3e370e75a"
format_stamp: "Formatted at 2026-09-26 07:34:09 on dist-test-slave-wdjc"
server_key: "b33698c1fe2c4d8127ab0576f4118073"
server_key_iv: "e882ba0141fd1c6c043ff6f05b54a485"
server_key_version: "encryptionkey@0"
I20260926 07:34:09.782919 9437 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 07:34:09.786242 9452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:09.787881 9437 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 07:34:09.788036 9437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407783529717-22743-0/minicluster-data/master-0/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407783529717-22743-0/minicluster-data/master-0/wal
uuid: "bd1b7a31c22d43919807b0d3e370e75a"
format_stamp: "Formatted at 2026-09-26 07:34:09 on dist-test-slave-wdjc"
server_key: "b33698c1fe2c4d8127ab0576f4118073"
server_key_iv: "e882ba0141fd1c6c043ff6f05b54a485"
server_key_version: "encryptionkey@0"
I20260926 07:34:09.788170 9437 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407783529717-22743-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407783529717-22743-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407783529717-22743-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:34:09.813055 9437 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:34:09.816934 9437 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:34:09.817121 9437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:34:09.825011 9437 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.254:33575
I20260926 07:34:09.825040 9504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.254:33575 every 8 connection(s)
I20260926 07:34:09.826121 9437 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
I20260926 07:34:09.829202 9505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:34:09.829725 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 9437
I20260926 07:34:09.829879 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407783529717-22743-0/minicluster-data/master-0/wal/instance
I20260926 07:34:09.830276 22743 external_mini_cluster.cc:1428] Setting key 991cb2ebd40667ab0d812f5cde3baa59
I20260926 07:34:09.836687 9505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd1b7a31c22d43919807b0d3e370e75a: Bootstrap starting.
I20260926 07:34:09.839285 9505 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd1b7a31c22d43919807b0d3e370e75a: Neither blocks nor log segments found. Creating new log.
Sep 26 07:34:09 dist-test-slave-wdjc krb5kdc[9424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408049, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.53.254@KRBTEST.COM
I20260926 07:34:09.840193 9505 log.cc:824] T 00000000000000000000000000000000 P bd1b7a31c22d43919807b0d3e370e75a: Log is configured to *not* fsync() on all Append() calls
I20260926 07:34:09.842929 9505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd1b7a31c22d43919807b0d3e370e75a: No bootstrap required, opened a new log
I20260926 07:34:09.846055 9508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:09.831897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58852 (local address 127.22.53.254:33575)
0926 07:34:09.832391 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:09.832410 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:09.832448 (+ 38us) server_negotiation.cc:407] Connection header received
0926 07:34:09.833240 (+ 792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:09.833261 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:09.833657 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:09.834033 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:09.834910 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:09.835958 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:09.837027 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:09.837524 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:09.837712 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:09.840455 (+ 2743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:09.840461 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:09.840476 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:09.840508 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:09.842840 (+ 2332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:09.843446 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:09.843453 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:09.843459 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:09.843593 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:09.843901 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:09.843904 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:09.843906 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:09.844344 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:09.844612 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:09.844915 (+ 303us) server_negotiation.cc:299] Negotiation successful
0926 07:34:09.845270 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":144,"threads_started":1}
I20260926 07:34:09.847108 9505 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd1b7a31c22d43919807b0d3e370e75a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd1b7a31c22d43919807b0d3e370e75a" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 33575 } }
I20260926 07:34:09.847442 9505 raft_consensus.cc:374] T 00000000000000000000000000000000 P bd1b7a31c22d43919807b0d3e370e75a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:34:09.847492 9505 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd1b7a31c22d43919807b0d3e370e75a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd1b7a31c22d43919807b0d3e370e75a, State: Initialized, Role: FOLLOWER
I20260926 07:34:09.847965 9505 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd1b7a31c22d43919807b0d3e370e75a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd1b7a31c22d43919807b0d3e370e75a" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 33575 } }
I20260926 07:34:09.848129 9505 raft_consensus.cc:388] T 00000000000000000000000000000000 P bd1b7a31c22d43919807b0d3e370e75a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:34:09.848192 9505 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd1b7a31c22d43919807b0d3e370e75a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:34:09.848304 9505 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd1b7a31c22d43919807b0d3e370e75a [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:34:09.849391 9505 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd1b7a31c22d43919807b0d3e370e75a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd1b7a31c22d43919807b0d3e370e75a" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 33575 } }
I20260926 07:34:09.849749 9505 leader_election.cc:302] T 00000000000000000000000000000000 P bd1b7a31c22d43919807b0d3e370e75a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd1b7a31c22d43919807b0d3e370e75a; no voters:
I20260926 07:34:09.850133 9505 leader_election.cc:288] T 00000000000000000000000000000000 P bd1b7a31c22d43919807b0d3e370e75a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:34:09.850234 9510 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd1b7a31c22d43919807b0d3e370e75a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:34:09.850440 9510 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd1b7a31c22d43919807b0d3e370e75a [term 1 LEADER]: Becoming Leader. State: Replica: bd1b7a31c22d43919807b0d3e370e75a, State: Running, Role: LEADER
I20260926 07:34:09.850808 9510 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd1b7a31c22d43919807b0d3e370e75a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd1b7a31c22d43919807b0d3e370e75a" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 33575 } }
I20260926 07:34:09.851519 9505 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd1b7a31c22d43919807b0d3e370e75a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:34:09.852700 9512 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd1b7a31c22d43919807b0d3e370e75a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bd1b7a31c22d43919807b0d3e370e75a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd1b7a31c22d43919807b0d3e370e75a" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 33575 } } }
I20260926 07:34:09.852846 9512 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd1b7a31c22d43919807b0d3e370e75a [sys.catalog]: This master's current role is: LEADER
I20260926 07:34:09.853185 9511 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd1b7a31c22d43919807b0d3e370e75a [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd1b7a31c22d43919807b0d3e370e75a. Latest consensus state: current_term: 1 leader_uuid: "bd1b7a31c22d43919807b0d3e370e75a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd1b7a31c22d43919807b0d3e370e75a" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 33575 } } }
I20260926 07:34:09.853278 9519 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:34:09.853345 9511 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd1b7a31c22d43919807b0d3e370e75a [sys.catalog]: This master's current role is: LEADER
I20260926 07:34:09.856310 9519 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:34:09.862473 9519 catalog_manager.cc:1347] Generated new cluster ID: 50ace99300b840e592dc3e2f9b43483c
I20260926 07:34:09.862563 9519 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:34:09.890101 9519 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:34:09.891134 9519 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:34:09.900493 9519 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bd1b7a31c22d43919807b0d3e370e75a: Generated new TSK 0
I20260926 07:34:09.901335 9519 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.53.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.53.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.53.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.193.keytab.
Entry for principal kudu/127.22.53.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.193.keytab.
I20260926 07:34:09.937327 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407783529717-22743-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407783529717-22743-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407783529717-22743-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407783529717-22743-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.53.193:0
--local_ip_for_outbound_sockets=127.22.53.193
--webserver_interface=127.22.53.193
--webserver_port=0
--tserver_master_addrs=127.22.53.254:33575
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.193.keytab
--principal=kudu/127.22.53.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:38951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:34:10.061401 9531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:34:10.061726 9531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:34:10.061775 9531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:34:10.066310 9531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:34:10.066406 9531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:34:10.066478 9531 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.53.193
I20260926 07:34:10.072376 9531 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:38951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407783529717-22743-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407783529717-22743-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.193.keytab
--principal=kudu/127.22.53.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.53.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407783529717-22743-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.53.193
--webserver_port=0
--tserver_master_addrs=127.22.53.254:33575
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9531
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.53.193
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407783529717-22743-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:34:10.073597 9531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:34:10.074743 9531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:34:10.085333 9536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:10.085296 9539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:10.085321 9537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:34:10.085721 9531 server_base.cc:1034] running on GCE node
I20260926 07:34:10.086179 9531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:34:10.086833 9531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:34:10.088007 9531 hybrid_clock.cc:648] HybridClock initialized: now 1790408050087986 us; error 39 us; skew 500 ppm
Sep 26 07:34:10 dist-test-slave-wdjc krb5kdc[9424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408050, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:34:10.091080 9531 init.cc:377] Logged in from keytab as kudu/127.22.53.193@KRBTEST.COM (short username kudu)
I20260926 07:34:10.092104 9531 webserver.cc:460] Webserver started at http://127.22.53.193:36935/ using document root <none> and password file <none>
I20260926 07:34:10.092865 9531 fs_manager.cc:359] Metadata directory not provided
I20260926 07:34:10.092970 9531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:34:10.093219 9531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:34:10.095558 9531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407783529717-22743-0/minicluster-data/ts-0/data/instance:
uuid: "e1b7d8a21ecd4935af4c2d3bc4340639"
format_stamp: "Formatted at 2026-09-26 07:34:10 on dist-test-slave-wdjc"
server_key: "1991cf1836413ab6e5ba971b60a27da9"
server_key_iv: "a671305cdd6eccde8290a78be9563058"
server_key_version: "encryptionkey@0"
I20260926 07:34:10.096187 9531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407783529717-22743-0/minicluster-data/ts-0/wal/instance:
uuid: "e1b7d8a21ecd4935af4c2d3bc4340639"
format_stamp: "Formatted at 2026-09-26 07:34:10 on dist-test-slave-wdjc"
server_key: "1991cf1836413ab6e5ba971b60a27da9"
server_key_iv: "a671305cdd6eccde8290a78be9563058"
server_key_version: "encryptionkey@0"
I20260926 07:34:10.100497 9531 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260926 07:34:10.103505 9546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:10.105074 9531 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:34:10.105247 9531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407783529717-22743-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407783529717-22743-0/minicluster-data/ts-0/wal
uuid: "e1b7d8a21ecd4935af4c2d3bc4340639"
format_stamp: "Formatted at 2026-09-26 07:34:10 on dist-test-slave-wdjc"
server_key: "1991cf1836413ab6e5ba971b60a27da9"
server_key_iv: "a671305cdd6eccde8290a78be9563058"
server_key_version: "encryptionkey@0"
I20260926 07:34:10.105396 9531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407783529717-22743-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407783529717-22743-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407783529717-22743-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:34:10.115877 9531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:34:10.119728 9531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:34:10.119978 9531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:34:10.120710 9531 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:34:10.121618 9531 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:34:10.121692 9531 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:10.121776 9531 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:34:10.121819 9531 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:10.131402 9531 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.193:37767
I20260926 07:34:10.131424 9659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.193:37767 every 8 connection(s)
I20260926 07:34:10.132393 9531 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407783529717-22743-0/minicluster-data/ts-0/data/info.pb
I20260926 07:34:10.135138 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 9531
I20260926 07:34:10.135293 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407783529717-22743-0/minicluster-data/ts-0/wal/instance
I20260926 07:34:10.135660 22743 external_mini_cluster.cc:1428] Setting key 33bbe5321c6b109ccf90bd314a885783
Sep 26 07:34:10 dist-test-slave-wdjc krb5kdc[9424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408050, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.193@KRBTEST.COM for kudu/127.22.53.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.22.53.194@KRBTEST.COM; defaulting to no policy
I20260926 07:34:10.148041 9508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:10.134525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.193:51727 (local address 127.22.53.254:33575)
0926 07:34:10.134693 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:10.134696 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:10.135602 (+ 906us) server_negotiation.cc:407] Connection header received
0926 07:34:10.137147 (+ 1545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:10.137151 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:10.137206 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:10.137291 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:10.138576 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:10.139118 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:10.140308 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:10.140734 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:10.140861 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:10.143830 (+ 2969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:10.143837 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:10.143843 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:10.143882 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:10.145814 (+ 1932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:10.146384 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:10.146388 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:10.146390 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:10.146494 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:10.146905 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:10.146908 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:10.146910 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:10.147111 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:10.147196 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:10.147735 (+ 539us) server_negotiation.cc:299] Negotiation successful
0926 07:34:10.147906 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 07:34:10.149230 9662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:10.134845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:33575 (local address 127.22.53.193:51727)
0926 07:34:10.135398 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:10.135464 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:10.136919 (+ 1455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:10.137490 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:10.137498 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:10.137895 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:10.138416 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:10.138430 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:10.139269 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:10.139273 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:10.140164 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:10.140184 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:10.140915 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:10.140918 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:10.141091 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:10.141769 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:10.141787 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:10.143615 (+ 1828us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:34:10.145962 (+ 2347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:10.145968 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:10.145984 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:10.146270 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:10.146616 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:10.146621 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:10.146623 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:10.146792 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:10.147209 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:34:10.147215 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:34:10.147496 (+ 281us) client_negotiation.cc:769] Sending connection context
0926 07:34:10.147667 (+ 171us) client_negotiation.cc:240] Negotiation successful
0926 07:34:10.148118 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":119,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.53.194@KRBTEST.COM" created.
I20260926 07:34:10.150776 9660 heartbeater.cc:347] Connected to a master server at 127.22.53.254:33575
I20260926 07:34:10.151059 9660 heartbeater.cc:464] Registering TS with master...
I20260926 07:34:10.151708 9660 heartbeater.cc:511] Master 127.22.53.254:33575 requested a full tablet report, sending...
I20260926 07:34:10.153667 9469 ts_manager.cc:194] Registered new tserver with Master: e1b7d8a21ecd4935af4c2d3bc4340639 (127.22.53.193:37767)
I20260926 07:34:10.155493 9469 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.53.193@KRBTEST.COM'} at 127.22.53.193:51727
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.53.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.194.keytab.
Entry for principal kudu/127.22.53.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.194.keytab.
I20260926 07:34:10.168155 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407783529717-22743-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407783529717-22743-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407783529717-22743-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407783529717-22743-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.53.194:0
--local_ip_for_outbound_sockets=127.22.53.194
--webserver_interface=127.22.53.194
--webserver_port=0
--tserver_master_addrs=127.22.53.254:33575
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.194.keytab
--principal=kudu/127.22.53.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:38951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:34:10.290940 9665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:34:10.291321 9665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:34:10.291370 9665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:34:10.295747 9665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:34:10.295831 9665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:34:10.295940 9665 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.53.194
I20260926 07:34:10.301904 9665 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:38951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407783529717-22743-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407783529717-22743-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.194.keytab
--principal=kudu/127.22.53.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.53.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407783529717-22743-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.53.194
--webserver_port=0
--tserver_master_addrs=127.22.53.254:33575
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9665
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.53.194
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407783529717-22743-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:34:10.303114 9665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:34:10.304234 9665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:34:10.315285 9670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:10.315459 9673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:10.315521 9671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:34:10.315758 9665 server_base.cc:1034] running on GCE node
I20260926 07:34:10.316231 9665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:34:10.317128 9665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:34:10.318344 9665 hybrid_clock.cc:648] HybridClock initialized: now 1790408050318325 us; error 37 us; skew 500 ppm
Sep 26 07:34:10 dist-test-slave-wdjc krb5kdc[9424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408050, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:34:10.322010 9665 init.cc:377] Logged in from keytab as kudu/127.22.53.194@KRBTEST.COM (short username kudu)
I20260926 07:34:10.323129 9665 webserver.cc:460] Webserver started at http://127.22.53.194:33769/ using document root <none> and password file <none>
I20260926 07:34:10.323822 9665 fs_manager.cc:359] Metadata directory not provided
I20260926 07:34:10.323927 9665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:34:10.324177 9665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:34:10.326632 9665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407783529717-22743-0/minicluster-data/ts-1/data/instance:
uuid: "8005b8af97544cc5a7e1a4f10ac77642"
format_stamp: "Formatted at 2026-09-26 07:34:10 on dist-test-slave-wdjc"
server_key: "ff75c85c947a04c493c8329690d63995"
server_key_iv: "0e4ae0158b9a02da78a59d8a038a6222"
server_key_version: "encryptionkey@0"
I20260926 07:34:10.327293 9665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407783529717-22743-0/minicluster-data/ts-1/wal/instance:
uuid: "8005b8af97544cc5a7e1a4f10ac77642"
format_stamp: "Formatted at 2026-09-26 07:34:10 on dist-test-slave-wdjc"
server_key: "ff75c85c947a04c493c8329690d63995"
server_key_iv: "0e4ae0158b9a02da78a59d8a038a6222"
server_key_version: "encryptionkey@0"
I20260926 07:34:10.331918 9665 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260926 07:34:10.334862 9680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:10.336414 9665 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 07:34:10.336611 9665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407783529717-22743-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407783529717-22743-0/minicluster-data/ts-1/wal
uuid: "8005b8af97544cc5a7e1a4f10ac77642"
format_stamp: "Formatted at 2026-09-26 07:34:10 on dist-test-slave-wdjc"
server_key: "ff75c85c947a04c493c8329690d63995"
server_key_iv: "0e4ae0158b9a02da78a59d8a038a6222"
server_key_version: "encryptionkey@0"
I20260926 07:34:10.336760 9665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407783529717-22743-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407783529717-22743-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407783529717-22743-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:34:10.352594 9665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:34:10.356407 9665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:34:10.356818 9665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:34:10.357546 9665 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:34:10.358412 9665 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:34:10.358458 9665 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:10.358500 9665 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:34:10.358515 9665 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:10.368002 9665 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.194:43385
I20260926 07:34:10.368021 9793 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.194:43385 every 8 connection(s)
I20260926 07:34:10.369048 9665 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407783529717-22743-0/minicluster-data/ts-1/data/info.pb
I20260926 07:34:10.376212 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 9665
I20260926 07:34:10.376332 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407783529717-22743-0/minicluster-data/ts-1/wal/instance
I20260926 07:34:10.376695 22743 external_mini_cluster.cc:1428] Setting key d55fe276be502eeeb9e218bcbafc13bf
Sep 26 07:34:10 dist-test-slave-wdjc krb5kdc[9424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408050, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.194@KRBTEST.COM for kudu/127.22.53.254@KRBTEST.COM
I20260926 07:34:10.385183 9508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:10.371089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.194:55769 (local address 127.22.53.254:33575)
0926 07:34:10.371272 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:10.371277 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:10.372149 (+ 872us) server_negotiation.cc:407] Connection header received
0926 07:34:10.373617 (+ 1468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:10.373620 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:10.373684 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:10.373771 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:10.375029 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:10.375590 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:10.376955 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:10.377253 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:10.377403 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:10.380641 (+ 3238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:10.380645 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:10.380654 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:10.380685 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:10.382431 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:10.383193 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:10.383196 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:10.383197 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:10.383341 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:10.383805 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:10.383807 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:10.383808 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:10.384015 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:10.384111 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:10.384892 (+ 781us) server_negotiation.cc:299] Negotiation successful
0926 07:34:10.385029 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 07:34:10.386492 9796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:10.371482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:33575 (local address 127.22.53.194:55769)
0926 07:34:10.371972 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:10.372017 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:10.373371 (+ 1354us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:10.373958 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:10.373965 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:10.374361 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:10.374869 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:10.374885 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:10.375712 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:10.375714 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:10.376824 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:10.376837 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:10.377451 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:10.377456 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:10.377666 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:10.378370 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:10.378388 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:10.380332 (+ 1944us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:34:10.382601 (+ 2269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:10.382611 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:10.382635 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:10.383069 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:10.383458 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:10.383463 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:10.383467 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:10.383690 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:10.384130 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:34:10.384139 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:34:10.384581 (+ 442us) client_negotiation.cc:769] Sending connection context
0926 07:34:10.384837 (+ 256us) client_negotiation.cc:240] Negotiation successful
0926 07:34:10.385243 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":122,"threads_started":1}
WARNING: no policy specified for kudu/127.22.53.195@KRBTEST.COM; defaulting to no policy
I20260926 07:34:10.387966 9794 heartbeater.cc:347] Connected to a master server at 127.22.53.254:33575
I20260926 07:34:10.388327 9794 heartbeater.cc:464] Registering TS with master...
I20260926 07:34:10.389014 9794 heartbeater.cc:511] Master 127.22.53.254:33575 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.53.195@KRBTEST.COM" created.
I20260926 07:34:10.390291 9469 ts_manager.cc:194] Registered new tserver with Master: 8005b8af97544cc5a7e1a4f10ac77642 (127.22.53.194:43385)
I20260926 07:34:10.390936 9469 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.53.194@KRBTEST.COM'} at 127.22.53.194:55769
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.53.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.195.keytab.
Entry for principal kudu/127.22.53.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.195.keytab.
I20260926 07:34:10.408593 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407783529717-22743-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407783529717-22743-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407783529717-22743-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407783529717-22743-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.53.195:0
--local_ip_for_outbound_sockets=127.22.53.195
--webserver_interface=127.22.53.195
--webserver_port=0
--tserver_master_addrs=127.22.53.254:33575
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.195.keytab
--principal=kudu/127.22.53.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:38951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:34:10.527067 9799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:34:10.527375 9799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:34:10.527421 9799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:34:10.532603 9799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:34:10.532680 9799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:34:10.532784 9799 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.53.195
I20260926 07:34:10.538589 9799 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:38951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407783529717-22743-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407783529717-22743-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.195.keytab
--principal=kudu/127.22.53.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.53.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407783529717-22743-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.53.195
--webserver_port=0
--tserver_master_addrs=127.22.53.254:33575
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9799
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.53.195
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407783529717-22743-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:34:10.539754 9799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:34:10.540901 9799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:34:10.550828 9807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:10.550841 9804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:10.550851 9805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:34:10.551105 9799 server_base.cc:1034] running on GCE node
I20260926 07:34:10.551723 9799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:34:10.552485 9799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:34:10.553750 9799 hybrid_clock.cc:648] HybridClock initialized: now 1790408050553697 us; error 49 us; skew 500 ppm
Sep 26 07:34:10 dist-test-slave-wdjc krb5kdc[9424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408050, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:34:10.557025 9799 init.cc:377] Logged in from keytab as kudu/127.22.53.195@KRBTEST.COM (short username kudu)
I20260926 07:34:10.558058 9799 webserver.cc:460] Webserver started at http://127.22.53.195:44745/ using document root <none> and password file <none>
I20260926 07:34:10.558710 9799 fs_manager.cc:359] Metadata directory not provided
I20260926 07:34:10.558782 9799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:34:10.559026 9799 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:34:10.561504 9799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407783529717-22743-0/minicluster-data/ts-2/data/instance:
uuid: "0baf4253ee0243d0bff3b41ef224c210"
format_stamp: "Formatted at 2026-09-26 07:34:10 on dist-test-slave-wdjc"
server_key: "1aaa6541ecca37ccfea2051b30201485"
server_key_iv: "139912a21f63d30e3523992e26c6dc9c"
server_key_version: "encryptionkey@0"
I20260926 07:34:10.562225 9799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407783529717-22743-0/minicluster-data/ts-2/wal/instance:
uuid: "0baf4253ee0243d0bff3b41ef224c210"
format_stamp: "Formatted at 2026-09-26 07:34:10 on dist-test-slave-wdjc"
server_key: "1aaa6541ecca37ccfea2051b30201485"
server_key_iv: "139912a21f63d30e3523992e26c6dc9c"
server_key_version: "encryptionkey@0"
I20260926 07:34:10.566627 9799 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260926 07:34:10.569455 9814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:10.571026 9799 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:34:10.571158 9799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407783529717-22743-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407783529717-22743-0/minicluster-data/ts-2/wal
uuid: "0baf4253ee0243d0bff3b41ef224c210"
format_stamp: "Formatted at 2026-09-26 07:34:10 on dist-test-slave-wdjc"
server_key: "1aaa6541ecca37ccfea2051b30201485"
server_key_iv: "139912a21f63d30e3523992e26c6dc9c"
server_key_version: "encryptionkey@0"
I20260926 07:34:10.571283 9799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407783529717-22743-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407783529717-22743-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407783529717-22743-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:34:10.603257 9799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:34:10.606773 9799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:34:10.606950 9799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:34:10.607563 9799 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:34:10.608448 9799 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:34:10.608580 9799 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:10.608655 9799 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:34:10.608687 9799 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:10.618793 9799 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.195:34149
I20260926 07:34:10.618813 9927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.195:34149 every 8 connection(s)
I20260926 07:34:10.619947 9799 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407783529717-22743-0/minicluster-data/ts-2/data/info.pb
I20260926 07:34:10.626430 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 9799
I20260926 07:34:10.626579 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407783529717-22743-0/minicluster-data/ts-2/wal/instance
I20260926 07:34:10.626894 22743 external_mini_cluster.cc:1428] Setting key 30804f6bc6e01de6d4882f311a0a3eaf
Sep 26 07:34:10 dist-test-slave-wdjc krb5kdc[9424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408050, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.195@KRBTEST.COM for kudu/127.22.53.254@KRBTEST.COM
I20260926 07:34:10.635775 9508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:10.622148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.195:40557 (local address 127.22.53.254:33575)
0926 07:34:10.622353 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:10.622357 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:10.623239 (+ 882us) server_negotiation.cc:407] Connection header received
0926 07:34:10.624774 (+ 1535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:10.624778 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:10.624853 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:10.624967 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:10.626319 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:10.627036 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:10.628647 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:10.628898 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:10.629024 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:10.631946 (+ 2922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:10.631950 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:10.631953 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:10.632003 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:10.633495 (+ 1492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:10.634128 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:10.634132 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:10.634133 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:10.634269 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:10.634669 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:10.634672 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:10.634673 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:10.634887 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:10.634983 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:10.635518 (+ 535us) server_negotiation.cc:299] Negotiation successful
0926 07:34:10.635640 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260926 07:34:10.636825 9930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:10.622545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:33575 (local address 127.22.53.195:40557)
0926 07:34:10.623053 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:10.623101 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:10.624457 (+ 1356us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:10.625164 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:10.625172 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:10.625567 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:10.626120 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:10.626143 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:10.627177 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:10.627181 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:10.628462 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:10.628474 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:10.629071 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:10.629076 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:10.629308 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:10.630019 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:10.630037 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:10.631680 (+ 1643us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:34:10.633644 (+ 1964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:10.633651 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:10.633670 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:10.633997 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:10.634386 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:10.634389 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:10.634390 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:10.634545 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:10.635002 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:34:10.635007 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:34:10.635292 (+ 285us) client_negotiation.cc:769] Sending connection context
0926 07:34:10.635465 (+ 173us) client_negotiation.cc:240] Negotiation successful
0926 07:34:10.635765 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":131,"threads_started":1}
I20260926 07:34:10.638193 9928 heartbeater.cc:347] Connected to a master server at 127.22.53.254:33575
I20260926 07:34:10.638463 9928 heartbeater.cc:464] Registering TS with master...
I20260926 07:34:10.639001 9928 heartbeater.cc:511] Master 127.22.53.254:33575 requested a full tablet report, sending...
I20260926 07:34:10.640836 9468 ts_manager.cc:194] Registered new tserver with Master: 0baf4253ee0243d0bff3b41ef224c210 (127.22.53.195:34149)
I20260926 07:34:10.641600 22743 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:34:10.641743 9468 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.53.195@KRBTEST.COM'} at 127.22.53.195:40557
I20260926 07:34:10.652755 9508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:10.643971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58856 (local address 127.22.53.254:33575)
0926 07:34:10.644135 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:10.644141 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:10.644315 (+ 174us) server_negotiation.cc:407] Connection header received
0926 07:34:10.644486 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:10.644489 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:10.644575 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:10.644648 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:10.645388 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:10.645916 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:10.646776 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:10.647010 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:10.647123 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:10.648646 (+ 1523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:10.648649 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:10.648652 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:10.648681 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:10.650645 (+ 1964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:10.651134 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:10.651138 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:10.651140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:10.651227 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:10.651594 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:10.651597 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:10.651598 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:10.651883 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:10.651998 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:10.652381 (+ 383us) server_negotiation.cc:299] Negotiation successful
0926 07:34:10.652535 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 07:34:10.656293 9468 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58856:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:34:10.659509 9468 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:34:10.676291 9942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:10.669867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.193:37767 (local address 127.0.0.1:50642)
0926 07:34:10.671071 (+ 1204us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:10.671085 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:10.671183 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:10.671757 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:10.671763 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:10.671787 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:34:10.671935 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:10.671941 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:10.673154 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:10.673158 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:10.674519 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:10.674529 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:10.675719 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:10.675722 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:10.675811 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:10.675844 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 07:34:10.675943 (+ 99us) client_negotiation.cc:240] Negotiation successful
0926 07:34:10.676106 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1141,"mutex_wait_us":39,"thread_start_us":101,"threads_started":1}
I20260926 07:34:10.676379 9943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:10.670646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50642 (local address 127.22.53.193:37767)
0926 07:34:10.671026 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:10.671035 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:10.671307 (+ 272us) server_negotiation.cc:407] Connection header received
0926 07:34:10.671394 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:10.671402 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:10.671596 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:10.671763 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:34:10.672065 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:10.673022 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:10.674649 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:10.675584 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:10.675701 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:10.675878 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:10.676010 (+ 132us) server_negotiation.cc:299] Negotiation successful
0926 07:34:10.676219 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":101,"threads_started":1}
I20260926 07:34:10.677165 9940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:10.670135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.194:43385 (local address 127.0.0.1:51962)
0926 07:34:10.671244 (+ 1109us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:10.671258 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:10.671350 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:10.672022 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:10.672025 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:10.672045 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:34:10.672167 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:10.672174 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:10.673626 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:10.673629 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:10.674997 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:10.675009 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:10.676715 (+ 1706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:10.676718 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:10.676821 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:10.676845 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 07:34:10.676952 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 07:34:10.677032 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1043,"mutex_wait_us":137,"thread_start_us":107,"threads_started":1}
I20260926 07:34:10.677484 9944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:10.670645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51962 (local address 127.22.53.194:43385)
0926 07:34:10.671232 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:10.671241 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:10.671349 (+ 108us) server_negotiation.cc:407] Connection header received
0926 07:34:10.671455 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:10.671463 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:10.671652 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:10.671809 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:34:10.672306 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:10.673485 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:10.675150 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:10.676574 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:10.676696 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:10.676844 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:10.676992 (+ 148us) server_negotiation.cc:299] Negotiation successful
0926 07:34:10.677100 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":484,"thread_start_us":100,"threads_started":1}
I20260926 07:34:10.678165 9939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:10.669848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.195:34149 (local address 127.0.0.1:37862)
0926 07:34:10.670678 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:10.670739 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:10.670888 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:10.671928 (+ 1040us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:10.671931 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:10.671946 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:34:10.672061 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:10.672073 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:10.673616 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:10.673619 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:10.675003 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:10.675009 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:10.677734 (+ 2725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:10.677737 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:10.677842 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:10.677864 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 07:34:10.677935 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 07:34:10.678005 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":655,"mutex_wait_us":50,"thread_start_us":78,"threads_started":1}
I20260926 07:34:10.678696 9941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:10.669997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37862 (local address 127.22.53.195:34149)
0926 07:34:10.671455 (+ 1458us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:10.671466 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:10.671485 (+ 19us) server_negotiation.cc:407] Connection header received
0926 07:34:10.671546 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:10.671551 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:10.671693 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:10.671829 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:34:10.672194 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:10.673440 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:10.676148 (+ 2708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:10.677420 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:10.677535 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:10.677673 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:10.678457 (+ 784us) server_negotiation.cc:299] Negotiation successful
0926 07:34:10.678562 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1354,"thread_start_us":123,"threads_started":1}
I20260926 07:34:10.679862 9594 tablet_service.cc:1467] Processing CreateTablet for tablet 8ea842bad5e9424aa014064ed014a417 (DEFAULT_TABLE table=test-table [id=441517a484724c3fa3915a2cbf4bf31e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:34:10.680210 9728 tablet_service.cc:1467] Processing CreateTablet for tablet 8ea842bad5e9424aa014064ed014a417 (DEFAULT_TABLE table=test-table [id=441517a484724c3fa3915a2cbf4bf31e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:34:10.681043 9594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ea842bad5e9424aa014064ed014a417. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:34:10.681234 9728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ea842bad5e9424aa014064ed014a417. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:34:10.681521 9862 tablet_service.cc:1467] Processing CreateTablet for tablet 8ea842bad5e9424aa014064ed014a417 (DEFAULT_TABLE table=test-table [id=441517a484724c3fa3915a2cbf4bf31e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:34:10.682529 9862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ea842bad5e9424aa014064ed014a417. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:34:10.688114 9945 tablet_bootstrap.cc:492] T 8ea842bad5e9424aa014064ed014a417 P e1b7d8a21ecd4935af4c2d3bc4340639: Bootstrap starting.
I20260926 07:34:10.690210 9947 tablet_bootstrap.cc:492] T 8ea842bad5e9424aa014064ed014a417 P 0baf4253ee0243d0bff3b41ef224c210: Bootstrap starting.
I20260926 07:34:10.690712 9945 tablet_bootstrap.cc:654] T 8ea842bad5e9424aa014064ed014a417 P e1b7d8a21ecd4935af4c2d3bc4340639: Neither blocks nor log segments found. Creating new log.
I20260926 07:34:10.691563 9945 log.cc:824] T 8ea842bad5e9424aa014064ed014a417 P e1b7d8a21ecd4935af4c2d3bc4340639: Log is configured to *not* fsync() on all Append() calls
I20260926 07:34:10.692258 9947 tablet_bootstrap.cc:654] T 8ea842bad5e9424aa014064ed014a417 P 0baf4253ee0243d0bff3b41ef224c210: Neither blocks nor log segments found. Creating new log.
I20260926 07:34:10.693809 9947 log.cc:824] T 8ea842bad5e9424aa014064ed014a417 P 0baf4253ee0243d0bff3b41ef224c210: Log is configured to *not* fsync() on all Append() calls
I20260926 07:34:10.694031 9945 tablet_bootstrap.cc:492] T 8ea842bad5e9424aa014064ed014a417 P e1b7d8a21ecd4935af4c2d3bc4340639: No bootstrap required, opened a new log
I20260926 07:34:10.694271 9945 ts_tablet_manager.cc:1397] T 8ea842bad5e9424aa014064ed014a417 P e1b7d8a21ecd4935af4c2d3bc4340639: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 07:34:10.694628 9946 tablet_bootstrap.cc:492] T 8ea842bad5e9424aa014064ed014a417 P 8005b8af97544cc5a7e1a4f10ac77642: Bootstrap starting.
I20260926 07:34:10.696126 9947 tablet_bootstrap.cc:492] T 8ea842bad5e9424aa014064ed014a417 P 0baf4253ee0243d0bff3b41ef224c210: No bootstrap required, opened a new log
I20260926 07:34:10.696331 9947 ts_tablet_manager.cc:1397] T 8ea842bad5e9424aa014064ed014a417 P 0baf4253ee0243d0bff3b41ef224c210: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260926 07:34:10.697201 9946 tablet_bootstrap.cc:654] T 8ea842bad5e9424aa014064ed014a417 P 8005b8af97544cc5a7e1a4f10ac77642: Neither blocks nor log segments found. Creating new log.
I20260926 07:34:10.698230 9945 raft_consensus.cc:348] T 8ea842bad5e9424aa014064ed014a417 P e1b7d8a21ecd4935af4c2d3bc4340639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0baf4253ee0243d0bff3b41ef224c210" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 34149 } } peers { permanent_uuid: "e1b7d8a21ecd4935af4c2d3bc4340639" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 37767 } } peers { permanent_uuid: "8005b8af97544cc5a7e1a4f10ac77642" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 43385 } }
I20260926 07:34:10.698531 9945 raft_consensus.cc:374] T 8ea842bad5e9424aa014064ed014a417 P e1b7d8a21ecd4935af4c2d3bc4340639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:34:10.698583 9945 raft_consensus.cc:729] T 8ea842bad5e9424aa014064ed014a417 P e1b7d8a21ecd4935af4c2d3bc4340639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1b7d8a21ecd4935af4c2d3bc4340639, State: Initialized, Role: FOLLOWER
I20260926 07:34:10.698640 9946 log.cc:824] T 8ea842bad5e9424aa014064ed014a417 P 8005b8af97544cc5a7e1a4f10ac77642: Log is configured to *not* fsync() on all Append() calls
I20260926 07:34:10.699076 9945 consensus_queue.cc:260] T 8ea842bad5e9424aa014064ed014a417 P e1b7d8a21ecd4935af4c2d3bc4340639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0baf4253ee0243d0bff3b41ef224c210" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 34149 } } peers { permanent_uuid: "e1b7d8a21ecd4935af4c2d3bc4340639" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 37767 } } peers { permanent_uuid: "8005b8af97544cc5a7e1a4f10ac77642" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 43385 } }
I20260926 07:34:10.699827 9660 heartbeater.cc:503] Master 127.22.53.254:33575 was elected leader, sending a full tablet report...
I20260926 07:34:10.700253 9945 ts_tablet_manager.cc:1428] T 8ea842bad5e9424aa014064ed014a417 P e1b7d8a21ecd4935af4c2d3bc4340639: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260926 07:34:10.701001 9946 tablet_bootstrap.cc:492] T 8ea842bad5e9424aa014064ed014a417 P 8005b8af97544cc5a7e1a4f10ac77642: No bootstrap required, opened a new log
I20260926 07:34:10.700891 9947 raft_consensus.cc:348] T 8ea842bad5e9424aa014064ed014a417 P 0baf4253ee0243d0bff3b41ef224c210 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0baf4253ee0243d0bff3b41ef224c210" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 34149 } } peers { permanent_uuid: "e1b7d8a21ecd4935af4c2d3bc4340639" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 37767 } } peers { permanent_uuid: "8005b8af97544cc5a7e1a4f10ac77642" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 43385 } }
I20260926 07:34:10.701174 9947 raft_consensus.cc:374] T 8ea842bad5e9424aa014064ed014a417 P 0baf4253ee0243d0bff3b41ef224c210 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:34:10.701233 9946 ts_tablet_manager.cc:1397] T 8ea842bad5e9424aa014064ed014a417 P 8005b8af97544cc5a7e1a4f10ac77642: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260926 07:34:10.701234 9947 raft_consensus.cc:729] T 8ea842bad5e9424aa014064ed014a417 P 0baf4253ee0243d0bff3b41ef224c210 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0baf4253ee0243d0bff3b41ef224c210, State: Initialized, Role: FOLLOWER
I20260926 07:34:10.701713 9947 consensus_queue.cc:260] T 8ea842bad5e9424aa014064ed014a417 P 0baf4253ee0243d0bff3b41ef224c210 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0baf4253ee0243d0bff3b41ef224c210" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 34149 } } peers { permanent_uuid: "e1b7d8a21ecd4935af4c2d3bc4340639" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 37767 } } peers { permanent_uuid: "8005b8af97544cc5a7e1a4f10ac77642" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 43385 } }
I20260926 07:34:10.702487 9928 heartbeater.cc:503] Master 127.22.53.254:33575 was elected leader, sending a full tablet report...
I20260926 07:34:10.702858 9947 ts_tablet_manager.cc:1428] T 8ea842bad5e9424aa014064ed014a417 P 0baf4253ee0243d0bff3b41ef224c210: Time spent starting tablet: real 0.006s user 0.004s sys 0.003s
I20260926 07:34:10.705232 9946 raft_consensus.cc:348] T 8ea842bad5e9424aa014064ed014a417 P 8005b8af97544cc5a7e1a4f10ac77642 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0baf4253ee0243d0bff3b41ef224c210" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 34149 } } peers { permanent_uuid: "e1b7d8a21ecd4935af4c2d3bc4340639" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 37767 } } peers { permanent_uuid: "8005b8af97544cc5a7e1a4f10ac77642" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 43385 } }
I20260926 07:34:10.705531 9946 raft_consensus.cc:374] T 8ea842bad5e9424aa014064ed014a417 P 8005b8af97544cc5a7e1a4f10ac77642 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:34:10.705623 9946 raft_consensus.cc:729] T 8ea842bad5e9424aa014064ed014a417 P 8005b8af97544cc5a7e1a4f10ac77642 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8005b8af97544cc5a7e1a4f10ac77642, State: Initialized, Role: FOLLOWER
I20260926 07:34:10.706120 9946 consensus_queue.cc:260] T 8ea842bad5e9424aa014064ed014a417 P 8005b8af97544cc5a7e1a4f10ac77642 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0baf4253ee0243d0bff3b41ef224c210" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 34149 } } peers { permanent_uuid: "e1b7d8a21ecd4935af4c2d3bc4340639" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 37767 } } peers { permanent_uuid: "8005b8af97544cc5a7e1a4f10ac77642" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 43385 } }
I20260926 07:34:10.706868 9794 heartbeater.cc:503] Master 127.22.53.254:33575 was elected leader, sending a full tablet report...
I20260926 07:34:10.707269 9946 ts_tablet_manager.cc:1428] T 8ea842bad5e9424aa014064ed014a417 P 8005b8af97544cc5a7e1a4f10ac77642: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 07:34:10.765694 9953 raft_consensus.cc:482] T 8ea842bad5e9424aa014064ed014a417 P 8005b8af97544cc5a7e1a4f10ac77642 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:34:10.765939 9953 raft_consensus.cc:504] T 8ea842bad5e9424aa014064ed014a417 P 8005b8af97544cc5a7e1a4f10ac77642 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0baf4253ee0243d0bff3b41ef224c210" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 34149 } } peers { permanent_uuid: "e1b7d8a21ecd4935af4c2d3bc4340639" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 37767 } } peers { permanent_uuid: "8005b8af97544cc5a7e1a4f10ac77642" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 43385 } }
I20260926 07:34:10.767202 9953 leader_election.cc:288] T 8ea842bad5e9424aa014064ed014a417 P 8005b8af97544cc5a7e1a4f10ac77642 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0baf4253ee0243d0bff3b41ef224c210 (127.22.53.195:34149), e1b7d8a21ecd4935af4c2d3bc4340639 (127.22.53.193:37767)
I20260926 07:34:10.771814 9941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:10.767685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.194:49585 (local address 127.22.53.195:34149)
0926 07:34:10.767817 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:10.767821 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:10.768220 (+ 399us) server_negotiation.cc:407] Connection header received
0926 07:34:10.768374 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:10.768377 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:10.768437 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:10.768538 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:34:10.768863 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:10.769514 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:10.770721 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:10.771137 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:10.771304 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:10.771357 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:10.771590 (+ 233us) server_negotiation.cc:299] Negotiation successful
0926 07:34:10.771677 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 07:34:10.771806 9943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:10.767738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.194:44119 (local address 127.22.53.193:37767)
0926 07:34:10.767878 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:10.767883 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:10.767902 (+ 19us) server_negotiation.cc:407] Connection header received
0926 07:34:10.767966 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:10.767970 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:10.768026 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:10.768124 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:34:10.768418 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:10.769156 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:10.770629 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:10.771103 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:10.771263 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:10.771329 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:10.771540 (+ 211us) server_negotiation.cc:299] Negotiation successful
0926 07:34:10.771638 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 07:34:10.771833 9796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:10.767552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.193:37767 (local address 127.22.53.194:44119)
0926 07:34:10.767727 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:10.767746 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:10.767867 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:10.768140 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:10.768143 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:10.768163 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:34:10.768297 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:10.768304 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:10.769325 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:10.769332 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:10.770493 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:10.770503 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:10.771253 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:10.771257 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:10.771376 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:10.771428 (+ 52us) client_negotiation.cc:769] Sending connection context
0926 07:34:10.771521 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 07:34:10.771651 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60,"mutex_wait_us":23}
I20260926 07:34:10.771849 9954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:10.767552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.195:34149 (local address 127.22.53.194:49585)
0926 07:34:10.768107 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:10.768119 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:10.768252 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:10.768584 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:10.768587 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:10.768605 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:34:10.768743 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:10.768750 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:10.769632 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:10.769639 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:10.770590 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:10.770600 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:10.771303 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:10.771307 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:10.771454 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:10.771472 (+ 18us) client_negotiation.cc:769] Sending connection context
0926 07:34:10.771558 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 07:34:10.771678 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"mutex_wait_us":241,"thread_start_us":174,"threads_started":1}
I20260926 07:34:10.772667 9614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ea842bad5e9424aa014064ed014a417" candidate_uuid: "8005b8af97544cc5a7e1a4f10ac77642" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1b7d8a21ecd4935af4c2d3bc4340639" is_pre_election: true
I20260926 07:34:10.772698 9882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ea842bad5e9424aa014064ed014a417" candidate_uuid: "8005b8af97544cc5a7e1a4f10ac77642" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0baf4253ee0243d0bff3b41ef224c210" is_pre_election: true
I20260926 07:34:10.773013 9614 raft_consensus.cc:2454] T 8ea842bad5e9424aa014064ed014a417 P e1b7d8a21ecd4935af4c2d3bc4340639 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8005b8af97544cc5a7e1a4f10ac77642 in term 0.
I20260926 07:34:10.773011 9882 raft_consensus.cc:2454] T 8ea842bad5e9424aa014064ed014a417 P 0baf4253ee0243d0bff3b41ef224c210 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8005b8af97544cc5a7e1a4f10ac77642 in term 0.
I20260926 07:34:10.773548 9684 leader_election.cc:302] T 8ea842bad5e9424aa014064ed014a417 P 8005b8af97544cc5a7e1a4f10ac77642 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0baf4253ee0243d0bff3b41ef224c210, 8005b8af97544cc5a7e1a4f10ac77642, e1b7d8a21ecd4935af4c2d3bc4340639; no voters:
I20260926 07:34:10.773887 9953 raft_consensus.cc:2781] T 8ea842bad5e9424aa014064ed014a417 P 8005b8af97544cc5a7e1a4f10ac77642 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:34:10.773974 9953 raft_consensus.cc:482] T 8ea842bad5e9424aa014064ed014a417 P 8005b8af97544cc5a7e1a4f10ac77642 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:34:10.774030 9953 raft_consensus.cc:3037] T 8ea842bad5e9424aa014064ed014a417 P 8005b8af97544cc5a7e1a4f10ac77642 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:34:10.775255 9953 raft_consensus.cc:504] T 8ea842bad5e9424aa014064ed014a417 P 8005b8af97544cc5a7e1a4f10ac77642 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0baf4253ee0243d0bff3b41ef224c210" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 34149 } } peers { permanent_uuid: "e1b7d8a21ecd4935af4c2d3bc4340639" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 37767 } } peers { permanent_uuid: "8005b8af97544cc5a7e1a4f10ac77642" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 43385 } }
I20260926 07:34:10.775738 9953 leader_election.cc:288] T 8ea842bad5e9424aa014064ed014a417 P 8005b8af97544cc5a7e1a4f10ac77642 [CANDIDATE]: Term 1 election: Requested vote from peers 0baf4253ee0243d0bff3b41ef224c210 (127.22.53.195:34149), e1b7d8a21ecd4935af4c2d3bc4340639 (127.22.53.193:37767)
I20260926 07:34:10.776136 9882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ea842bad5e9424aa014064ed014a417" candidate_uuid: "8005b8af97544cc5a7e1a4f10ac77642" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0baf4253ee0243d0bff3b41ef224c210"
I20260926 07:34:10.776176 9614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ea842bad5e9424aa014064ed014a417" candidate_uuid: "8005b8af97544cc5a7e1a4f10ac77642" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1b7d8a21ecd4935af4c2d3bc4340639"
I20260926 07:34:10.776314 9614 raft_consensus.cc:3037] T 8ea842bad5e9424aa014064ed014a417 P e1b7d8a21ecd4935af4c2d3bc4340639 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:34:10.776324 9882 raft_consensus.cc:3037] T 8ea842bad5e9424aa014064ed014a417 P 0baf4253ee0243d0bff3b41ef224c210 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:34:10.777565 9614 raft_consensus.cc:2454] T 8ea842bad5e9424aa014064ed014a417 P e1b7d8a21ecd4935af4c2d3bc4340639 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8005b8af97544cc5a7e1a4f10ac77642 in term 1.
I20260926 07:34:10.777673 9882 raft_consensus.cc:2454] T 8ea842bad5e9424aa014064ed014a417 P 0baf4253ee0243d0bff3b41ef224c210 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8005b8af97544cc5a7e1a4f10ac77642 in term 1.
I20260926 07:34:10.777997 9681 leader_election.cc:302] T 8ea842bad5e9424aa014064ed014a417 P 8005b8af97544cc5a7e1a4f10ac77642 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8005b8af97544cc5a7e1a4f10ac77642, e1b7d8a21ecd4935af4c2d3bc4340639; no voters:
I20260926 07:34:10.778250 9953 raft_consensus.cc:2781] T 8ea842bad5e9424aa014064ed014a417 P 8005b8af97544cc5a7e1a4f10ac77642 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:34:10.778522 9953 raft_consensus.cc:686] T 8ea842bad5e9424aa014064ed014a417 P 8005b8af97544cc5a7e1a4f10ac77642 [term 1 LEADER]: Becoming Leader. State: Replica: 8005b8af97544cc5a7e1a4f10ac77642, State: Running, Role: LEADER
I20260926 07:34:10.778965 9953 consensus_queue.cc:237] T 8ea842bad5e9424aa014064ed014a417 P 8005b8af97544cc5a7e1a4f10ac77642 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0baf4253ee0243d0bff3b41ef224c210" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 34149 } } peers { permanent_uuid: "e1b7d8a21ecd4935af4c2d3bc4340639" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 37767 } } peers { permanent_uuid: "8005b8af97544cc5a7e1a4f10ac77642" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 43385 } }
I20260926 07:34:10.782591 9468 catalog_manager.cc:5495] T 8ea842bad5e9424aa014064ed014a417 P 8005b8af97544cc5a7e1a4f10ac77642 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8005b8af97544cc5a7e1a4f10ac77642 (127.22.53.194). New cstate: current_term: 1 leader_uuid: "8005b8af97544cc5a7e1a4f10ac77642" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0baf4253ee0243d0bff3b41ef224c210" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 34149 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1b7d8a21ecd4935af4c2d3bc4340639" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 37767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8005b8af97544cc5a7e1a4f10ac77642" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 43385 } health_report { overall_health: HEALTHY } } }
I20260926 07:34:10.803115 9944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:10.799341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51970 (local address 127.22.53.194:43385)
0926 07:34:10.799474 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:10.799478 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:10.799500 (+ 22us) server_negotiation.cc:407] Connection header received
0926 07:34:10.799582 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:10.799584 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:10.799653 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:10.799747 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:34:10.800046 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:10.800630 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:10.801582 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:10.801818 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:10.801923 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:10.802262 (+ 339us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:34:10.802708 (+ 446us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:34:10.802801 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:10.802918 (+ 117us) server_negotiation.cc:299] Negotiation successful
0926 07:34:10.802973 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 07:34:10.809530 9882 raft_consensus.cc:1261] T 8ea842bad5e9424aa014064ed014a417 P 0baf4253ee0243d0bff3b41ef224c210 [term 1 FOLLOWER]: Refusing update from remote peer 8005b8af97544cc5a7e1a4f10ac77642: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:34:10.809573 9614 raft_consensus.cc:1261] T 8ea842bad5e9424aa014064ed014a417 P e1b7d8a21ecd4935af4c2d3bc4340639 [term 1 FOLLOWER]: Refusing update from remote peer 8005b8af97544cc5a7e1a4f10ac77642: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:34:10.810328 9955 consensus_queue.cc:1035] T 8ea842bad5e9424aa014064ed014a417 P 8005b8af97544cc5a7e1a4f10ac77642 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0baf4253ee0243d0bff3b41ef224c210" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 34149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:34:10.810585 9953 consensus_queue.cc:1035] T 8ea842bad5e9424aa014064ed014a417 P 8005b8af97544cc5a7e1a4f10ac77642 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1b7d8a21ecd4935af4c2d3bc4340639" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 37767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:34:10.829424 9468 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58856:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:34:10.829635 9468 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58856:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:34:10.832681 9468 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bd1b7a31c22d43919807b0d3e370e75a: Sending DeleteTablet for 3 replicas of tablet 8ea842bad5e9424aa014064ed014a417
I20260926 07:34:10.833576 9862 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ea842bad5e9424aa014064ed014a417 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:34:10 UTC) from {username='kudu', principal='kudu/127.22.53.254@KRBTEST.COM'} at 127.0.0.1:37862
I20260926 07:34:10.833590 9594 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ea842bad5e9424aa014064ed014a417 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:34:10 UTC) from {username='kudu', principal='kudu/127.22.53.254@KRBTEST.COM'} at 127.0.0.1:50642
I20260926 07:34:10.834282 9969 tablet_replica.cc:330] T 8ea842bad5e9424aa014064ed014a417 P 0baf4253ee0243d0bff3b41ef224c210: stopping tablet replica
I20260926 07:34:10.834589 9728 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ea842bad5e9424aa014064ed014a417 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:34:10 UTC) from {username='kudu', principal='kudu/127.22.53.254@KRBTEST.COM'} at 127.0.0.1:51962
I20260926 07:34:10.834668 9969 raft_consensus.cc:2229] T 8ea842bad5e9424aa014064ed014a417 P 0baf4253ee0243d0bff3b41ef224c210 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:34:10.835091 9969 raft_consensus.cc:2258] T 8ea842bad5e9424aa014064ed014a417 P 0baf4253ee0243d0bff3b41ef224c210 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:34:10.835510 9970 tablet_replica.cc:330] T 8ea842bad5e9424aa014064ed014a417 P 8005b8af97544cc5a7e1a4f10ac77642: stopping tablet replica
I20260926 07:34:10.835847 9970 raft_consensus.cc:2229] T 8ea842bad5e9424aa014064ed014a417 P 8005b8af97544cc5a7e1a4f10ac77642 [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:34:10.836402 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 9531
I20260926 07:34:10.838675 9970 raft_consensus.cc:2258] T 8ea842bad5e9424aa014064ed014a417 P 8005b8af97544cc5a7e1a4f10ac77642 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:34:10.839356 9969 ts_tablet_manager.cc:1905] T 8ea842bad5e9424aa014064ed014a417 P 0baf4253ee0243d0bff3b41ef224c210: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:34:10.840040 9970 ts_tablet_manager.cc:1905] T 8ea842bad5e9424aa014064ed014a417 P 8005b8af97544cc5a7e1a4f10ac77642: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:34:10.842832 9969 ts_tablet_manager.cc:1918] T 8ea842bad5e9424aa014064ed014a417 P 0baf4253ee0243d0bff3b41ef224c210: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:34:10.842968 9970 ts_tablet_manager.cc:1918] T 8ea842bad5e9424aa014064ed014a417 P 8005b8af97544cc5a7e1a4f10ac77642: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:34:10.843077 9970 log.cc:1196] T 8ea842bad5e9424aa014064ed014a417 P 8005b8af97544cc5a7e1a4f10ac77642: Deleting WAL directory at /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407783529717-22743-0/minicluster-data/ts-1/wal/wals/8ea842bad5e9424aa014064ed014a417
I20260926 07:34:10.843081 9969 log.cc:1196] T 8ea842bad5e9424aa014064ed014a417 P 0baf4253ee0243d0bff3b41ef224c210: Deleting WAL directory at /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407783529717-22743-0/minicluster-data/ts-2/wal/wals/8ea842bad5e9424aa014064ed014a417
I20260926 07:34:10.843430 9970 ts_tablet_manager.cc:1939] T 8ea842bad5e9424aa014064ed014a417 P 8005b8af97544cc5a7e1a4f10ac77642: Deleting consensus metadata
I20260926 07:34:10.843621 9969 ts_tablet_manager.cc:1939] T 8ea842bad5e9424aa014064ed014a417 P 0baf4253ee0243d0bff3b41ef224c210: Deleting consensus metadata
I20260926 07:34:10.845057 9456 catalog_manager.cc:4841] TS 0baf4253ee0243d0bff3b41ef224c210 (127.22.53.195:34149): tablet 8ea842bad5e9424aa014064ed014a417 (table test-table [id=441517a484724c3fa3915a2cbf4bf31e]) successfully deleted
I20260926 07:34:10.845062 9454 catalog_manager.cc:4841] TS 8005b8af97544cc5a7e1a4f10ac77642 (127.22.53.194:43385): tablet 8ea842bad5e9424aa014064ed014a417 (table test-table [id=441517a484724c3fa3915a2cbf4bf31e]) successfully deleted
W20260926 07:34:10.847599 9453 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.53.193:37767 (error 108)
I20260926 07:34:10.848214 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 9665
I20260926 07:34:10.853541 9942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:10.853085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.193:37767 (local address 127.0.0.1:50656)
0926 07:34:10.853270 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:10.853379 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.193:37767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260926 07:34:10.853865 9453 catalog_manager.cc:4584] TS e1b7d8a21ecd4935af4c2d3bc4340639 (127.22.53.193:37767): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8ea842bad5e9424aa014064ed014a417: Network error: Client connection negotiation failed: client connection to 127.22.53.193:37767: connect: Connection refused (error 111)
I20260926 07:34:10.856066 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 9799
I20260926 07:34:10.862251 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 9437
2026-09-26T07:34:10Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2467 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-26T07:34:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:34:10Z Disabled control of system clock
I20260926 07:34:10.894461 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407783529717-22743-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407783529717-22743-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407783529717-22743-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.53.254:41839
--webserver_interface=127.22.53.254
--webserver_port=0
--builtin_ntp_servers=127.22.53.212:37611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.53.254:41839
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 07:34:11.009626 9975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:34:11.010013 9975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:34:11.010056 9975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:34:11.014237 9975 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:34:11.014310 9975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:34:11.014326 9975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:34:11.014343 9975 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:34:11.014359 9975 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:34:11.019423 9975 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:37611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407783529717-22743-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407783529717-22743-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.53.254:41839
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.53.254:41839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.53.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9975
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407783529717-22743-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:34:11.020742 9975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:34:11.022262 9975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:34:11.031356 9981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:11.031347 9980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:11.031347 9983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:34:11.031450 9975 server_base.cc:1034] running on GCE node
I20260926 07:34:11.032336 9975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:34:11.033758 9975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:34:11.034974 9975 hybrid_clock.cc:648] HybridClock initialized: now 1790408051034960 us; error 32 us; skew 500 ppm
I20260926 07:34:11.037199 9975 webserver.cc:460] Webserver started at http://127.22.53.254:41655/ using document root <none> and password file <none>
I20260926 07:34:11.037858 9975 fs_manager.cc:359] Metadata directory not provided
I20260926 07:34:11.037962 9975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:34:11.038211 9975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:34:11.040484 9975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407783529717-22743-0/minicluster-data/master-0/data/instance:
uuid: "56c23671a80d4ad3a8a85913f6dca39e"
format_stamp: "Formatted at 2026-09-26 07:34:11 on dist-test-slave-wdjc"
server_key: "94c7145dd56d765b6622b6f7970f6ad2"
server_key_iv: "91ec8803b5e72d2083f8f32e77ef6ffd"
server_key_version: "encryptionkey@0"
I20260926 07:34:11.041195 9975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407783529717-22743-0/minicluster-data/master-0/wal/instance:
uuid: "56c23671a80d4ad3a8a85913f6dca39e"
format_stamp: "Formatted at 2026-09-26 07:34:11 on dist-test-slave-wdjc"
server_key: "94c7145dd56d765b6622b6f7970f6ad2"
server_key_iv: "91ec8803b5e72d2083f8f32e77ef6ffd"
server_key_version: "encryptionkey@0"
I20260926 07:34:11.045295 9975 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 07:34:11.048110 9989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:11.049649 9975 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 07:34:11.049844 9975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407783529717-22743-0/minicluster-data/master-0/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407783529717-22743-0/minicluster-data/master-0/wal
uuid: "56c23671a80d4ad3a8a85913f6dca39e"
format_stamp: "Formatted at 2026-09-26 07:34:11 on dist-test-slave-wdjc"
server_key: "94c7145dd56d765b6622b6f7970f6ad2"
server_key_iv: "91ec8803b5e72d2083f8f32e77ef6ffd"
server_key_version: "encryptionkey@0"
I20260926 07:34:11.049993 9975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407783529717-22743-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407783529717-22743-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407783529717-22743-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:34:11.067843 9975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:34:11.068769 9975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:34:11.069010 9975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:34:11.076184 9975 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.254:41839
I20260926 07:34:11.076244 10041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.254:41839 every 8 connection(s)
I20260926 07:34:11.077311 9975 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
I20260926 07:34:11.080821 10042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:34:11.081231 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 9975
I20260926 07:34:11.081364 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407783529717-22743-0/minicluster-data/master-0/wal/instance
I20260926 07:34:11.081719 22743 external_mini_cluster.cc:1428] Setting key beed3e77ff475c714c089cddbd2540f8
I20260926 07:34:11.087601 10042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56c23671a80d4ad3a8a85913f6dca39e: Bootstrap starting.
I20260926 07:34:11.090337 10042 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 56c23671a80d4ad3a8a85913f6dca39e: Neither blocks nor log segments found. Creating new log.
I20260926 07:34:11.091288 10042 log.cc:824] T 00000000000000000000000000000000 P 56c23671a80d4ad3a8a85913f6dca39e: Log is configured to *not* fsync() on all Append() calls
I20260926 07:34:11.091996 10045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:11.083297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43204 (local address 127.22.53.254:41839)
0926 07:34:11.084015 (+ 718us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:11.084030 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:11.084067 (+ 37us) server_negotiation.cc:407] Connection header received
0926 07:34:11.084893 (+ 826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:11.084927 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:11.085382 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:11.085765 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:11.086262 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:11.087272 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:11.088288 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:11.088873 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:11.089066 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:11.090028 (+ 962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:11.090033 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:11.090048 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:34:11.090502 (+ 454us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:11.090580 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:34:11.090609 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:11.090719 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:11.090885 (+ 166us) server_negotiation.cc:299] Negotiation successful
0926 07:34:11.091168 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":133,"threads_started":1}
I20260926 07:34:11.093494 10042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56c23671a80d4ad3a8a85913f6dca39e: No bootstrap required, opened a new log
I20260926 07:34:11.097375 10042 raft_consensus.cc:348] T 00000000000000000000000000000000 P 56c23671a80d4ad3a8a85913f6dca39e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56c23671a80d4ad3a8a85913f6dca39e" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 41839 } }
I20260926 07:34:11.097676 10042 raft_consensus.cc:374] T 00000000000000000000000000000000 P 56c23671a80d4ad3a8a85913f6dca39e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:34:11.097733 10042 raft_consensus.cc:729] T 00000000000000000000000000000000 P 56c23671a80d4ad3a8a85913f6dca39e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56c23671a80d4ad3a8a85913f6dca39e, State: Initialized, Role: FOLLOWER
I20260926 07:34:11.098253 10042 consensus_queue.cc:260] T 00000000000000000000000000000000 P 56c23671a80d4ad3a8a85913f6dca39e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56c23671a80d4ad3a8a85913f6dca39e" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 41839 } }
I20260926 07:34:11.098436 10042 raft_consensus.cc:388] T 00000000000000000000000000000000 P 56c23671a80d4ad3a8a85913f6dca39e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:34:11.098526 10042 raft_consensus.cc:482] T 00000000000000000000000000000000 P 56c23671a80d4ad3a8a85913f6dca39e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:34:11.098655 10042 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 56c23671a80d4ad3a8a85913f6dca39e [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:34:11.099697 10042 raft_consensus.cc:504] T 00000000000000000000000000000000 P 56c23671a80d4ad3a8a85913f6dca39e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56c23671a80d4ad3a8a85913f6dca39e" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 41839 } }
I20260926 07:34:11.100031 10042 leader_election.cc:302] T 00000000000000000000000000000000 P 56c23671a80d4ad3a8a85913f6dca39e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 56c23671a80d4ad3a8a85913f6dca39e; no voters:
I20260926 07:34:11.100337 10042 leader_election.cc:288] T 00000000000000000000000000000000 P 56c23671a80d4ad3a8a85913f6dca39e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:34:11.100500 10047 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 56c23671a80d4ad3a8a85913f6dca39e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:34:11.100800 10047 raft_consensus.cc:686] T 00000000000000000000000000000000 P 56c23671a80d4ad3a8a85913f6dca39e [term 1 LEADER]: Becoming Leader. State: Replica: 56c23671a80d4ad3a8a85913f6dca39e, State: Running, Role: LEADER
I20260926 07:34:11.101138 10047 consensus_queue.cc:237] T 00000000000000000000000000000000 P 56c23671a80d4ad3a8a85913f6dca39e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56c23671a80d4ad3a8a85913f6dca39e" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 41839 } }
I20260926 07:34:11.101696 10042 sys_catalog.cc:564] T 00000000000000000000000000000000 P 56c23671a80d4ad3a8a85913f6dca39e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:34:11.102924 10049 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56c23671a80d4ad3a8a85913f6dca39e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 56c23671a80d4ad3a8a85913f6dca39e. Latest consensus state: current_term: 1 leader_uuid: "56c23671a80d4ad3a8a85913f6dca39e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56c23671a80d4ad3a8a85913f6dca39e" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 41839 } } }
I20260926 07:34:11.103062 10049 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56c23671a80d4ad3a8a85913f6dca39e [sys.catalog]: This master's current role is: LEADER
I20260926 07:34:11.103335 10048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56c23671a80d4ad3a8a85913f6dca39e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "56c23671a80d4ad3a8a85913f6dca39e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56c23671a80d4ad3a8a85913f6dca39e" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 41839 } } }
I20260926 07:34:11.103428 10048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56c23671a80d4ad3a8a85913f6dca39e [sys.catalog]: This master's current role is: LEADER
I20260926 07:34:11.103539 10056 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:34:11.106769 10056 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:34:11.112787 10056 catalog_manager.cc:1347] Generated new cluster ID: d06bd98dc3064071a63e32b692aa35d0
I20260926 07:34:11.112883 10056 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:34:11.120679 10056 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:34:11.121824 10056 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:34:11.130038 10056 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 56c23671a80d4ad3a8a85913f6dca39e: Generated new TSK 0
I20260926 07:34:11.130882 10056 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:34:11.139393 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407783529717-22743-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407783529717-22743-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407783529717-22743-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407783529717-22743-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.53.193:0
--local_ip_for_outbound_sockets=127.22.53.193
--webserver_interface=127.22.53.193
--webserver_port=0
--tserver_master_addrs=127.22.53.254:41839
--builtin_ntp_servers=127.22.53.212:37611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 07:34:11.263001 10066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:34:11.263347 10066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:34:11.263424 10066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:34:11.267767 10066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:34:11.267874 10066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:34:11.267974 10066 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.53.193
I20260926 07:34:11.273455 10066 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:37611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407783529717-22743-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407783529717-22743-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.53.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407783529717-22743-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.53.193
--webserver_port=0
--tserver_master_addrs=127.22.53.254:41839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10066
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.53.193
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407783529717-22743-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:34:11.275022 10066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:34:11.276479 10066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:34:11.286882 10071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:11.286984 10072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:11.286875 10074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:34:11.287388 10066 server_base.cc:1034] running on GCE node
I20260926 07:34:11.287901 10066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:34:11.288694 10066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:34:11.289916 10066 hybrid_clock.cc:648] HybridClock initialized: now 1790408051289880 us; error 52 us; skew 500 ppm
I20260926 07:34:11.292044 10066 webserver.cc:460] Webserver started at http://127.22.53.193:36707/ using document root <none> and password file <none>
I20260926 07:34:11.292801 10066 fs_manager.cc:359] Metadata directory not provided
I20260926 07:34:11.292907 10066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:34:11.293159 10066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:34:11.295506 10066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407783529717-22743-0/minicluster-data/ts-0/data/instance:
uuid: "166722ce087b49c293c6a941db6cc467"
format_stamp: "Formatted at 2026-09-26 07:34:11 on dist-test-slave-wdjc"
server_key: "ad8c82a68bed99f2bef73c92a22fe46d"
server_key_iv: "a8557699ea809cb97f6dce11224565ce"
server_key_version: "encryptionkey@0"
I20260926 07:34:11.296141 10066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407783529717-22743-0/minicluster-data/ts-0/wal/instance:
uuid: "166722ce087b49c293c6a941db6cc467"
format_stamp: "Formatted at 2026-09-26 07:34:11 on dist-test-slave-wdjc"
server_key: "ad8c82a68bed99f2bef73c92a22fe46d"
server_key_iv: "a8557699ea809cb97f6dce11224565ce"
server_key_version: "encryptionkey@0"
I20260926 07:34:11.300446 10066 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 07:34:11.303368 10080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:11.304714 10066 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 07:34:11.304863 10066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407783529717-22743-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407783529717-22743-0/minicluster-data/ts-0/wal
uuid: "166722ce087b49c293c6a941db6cc467"
format_stamp: "Formatted at 2026-09-26 07:34:11 on dist-test-slave-wdjc"
server_key: "ad8c82a68bed99f2bef73c92a22fe46d"
server_key_iv: "a8557699ea809cb97f6dce11224565ce"
server_key_version: "encryptionkey@0"
I20260926 07:34:11.305004 10066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407783529717-22743-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407783529717-22743-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407783529717-22743-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:34:11.327189 10066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:34:11.327984 10066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:34:11.328227 10066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:34:11.328917 10066 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:34:11.330004 10066 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:34:11.330082 10066 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:11.330163 10066 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:34:11.330214 10066 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:11.339784 10066 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.193:43629
I20260926 07:34:11.339844 10193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.193:43629 every 8 connection(s)
I20260926 07:34:11.340956 10066 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407783529717-22743-0/minicluster-data/ts-0/data/info.pb
I20260926 07:34:11.348014 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 10066
I20260926 07:34:11.348228 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407783529717-22743-0/minicluster-data/ts-0/wal/instance
I20260926 07:34:11.348639 22743 external_mini_cluster.cc:1428] Setting key 87a6a88ca1c7b3d894dd16b88805ce47
I20260926 07:34:11.351173 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407783529717-22743-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407783529717-22743-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407783529717-22743-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407783529717-22743-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.53.194:0
--local_ip_for_outbound_sockets=127.22.53.194
--webserver_interface=127.22.53.194
--webserver_port=0
--tserver_master_addrs=127.22.53.254:41839
--builtin_ntp_servers=127.22.53.212:37611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 07:34:11.352980 10045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:11.343258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.193:49669 (local address 127.22.53.254:41839)
0926 07:34:11.343431 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:11.343435 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:11.344863 (+ 1428us) server_negotiation.cc:407] Connection header received
0926 07:34:11.346463 (+ 1600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:11.346467 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:11.346533 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:11.346640 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:11.347762 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:11.348723 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:11.350368 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:11.350649 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:11.350778 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:11.352362 (+ 1584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:11.352365 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:11.352368 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:34:11.352382 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:11.352404 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:34:11.352412 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:11.352514 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:11.352739 (+ 225us) server_negotiation.cc:299] Negotiation successful
0926 07:34:11.352824 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 07:34:11.354094 10196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:11.343746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:41839 (local address 127.22.53.193:49669)
0926 07:34:11.344672 (+ 926us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:11.344733 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:11.346192 (+ 1459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:11.346859 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:11.346868 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:11.346971 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:11.347552 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:11.347567 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:11.348885 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:11.348889 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:11.350217 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:11.350232 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:11.350841 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:11.350846 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:11.351039 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:11.351936 (+ 897us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:34:11.351937 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:11.351988 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:34:11.352070 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:34:11.352518 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:34:11.352555 (+ 37us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:34:11.352563 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 07:34:11.352740 (+ 177us) client_negotiation.cc:240] Negotiation successful
0926 07:34:11.353096 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":126,"threads_started":1}
I20260926 07:34:11.355638 10194 heartbeater.cc:347] Connected to a master server at 127.22.53.254:41839
I20260926 07:34:11.355953 10194 heartbeater.cc:464] Registering TS with master...
I20260926 07:34:11.356801 10194 heartbeater.cc:511] Master 127.22.53.254:41839 requested a full tablet report, sending...
I20260926 07:34:11.359015 10002 ts_manager.cc:194] Registered new tserver with Master: 166722ce087b49c293c6a941db6cc467 (127.22.53.193:43629)
I20260926 07:34:11.361008 10002 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.53.193:49669
W20260926 07:34:11.471371 10197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:34:11.471693 10197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:34:11.471743 10197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:34:11.476035 10197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:34:11.476122 10197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:34:11.476182 10197 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.53.194
I20260926 07:34:11.481168 10197 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:37611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407783529717-22743-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407783529717-22743-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.53.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407783529717-22743-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.53.194
--webserver_port=0
--tserver_master_addrs=127.22.53.254:41839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10197
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.53.194
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407783529717-22743-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:34:11.482331 10197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:34:11.483937 10197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:34:11.494119 10202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:11.494130 10203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:11.494129 10205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:34:11.494426 10197 server_base.cc:1034] running on GCE node
I20260926 07:34:11.494935 10197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:34:11.495647 10197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:34:11.496917 10197 hybrid_clock.cc:648] HybridClock initialized: now 1790408051496903 us; error 36 us; skew 500 ppm
I20260926 07:34:11.499110 10197 webserver.cc:460] Webserver started at http://127.22.53.194:46067/ using document root <none> and password file <none>
I20260926 07:34:11.499766 10197 fs_manager.cc:359] Metadata directory not provided
I20260926 07:34:11.499867 10197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:34:11.500118 10197 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:34:11.502992 10197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407783529717-22743-0/minicluster-data/ts-1/data/instance:
uuid: "435c5eed72a44782be096e57d3847ca0"
format_stamp: "Formatted at 2026-09-26 07:34:11 on dist-test-slave-wdjc"
server_key: "1061675f62a9c6a27c737099fa921061"
server_key_iv: "149337e2e470e31dc62eab7edea6345d"
server_key_version: "encryptionkey@0"
I20260926 07:34:11.503681 10197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407783529717-22743-0/minicluster-data/ts-1/wal/instance:
uuid: "435c5eed72a44782be096e57d3847ca0"
format_stamp: "Formatted at 2026-09-26 07:34:11 on dist-test-slave-wdjc"
server_key: "1061675f62a9c6a27c737099fa921061"
server_key_iv: "149337e2e470e31dc62eab7edea6345d"
server_key_version: "encryptionkey@0"
I20260926 07:34:11.507949 10197 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260926 07:34:11.511080 10211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:11.512794 10197 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 07:34:11.512951 10197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407783529717-22743-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407783529717-22743-0/minicluster-data/ts-1/wal
uuid: "435c5eed72a44782be096e57d3847ca0"
format_stamp: "Formatted at 2026-09-26 07:34:11 on dist-test-slave-wdjc"
server_key: "1061675f62a9c6a27c737099fa921061"
server_key_iv: "149337e2e470e31dc62eab7edea6345d"
server_key_version: "encryptionkey@0"
I20260926 07:34:11.513092 10197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407783529717-22743-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407783529717-22743-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407783529717-22743-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:34:11.540863 10197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:34:11.541674 10197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:34:11.541919 10197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:34:11.542573 10197 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:34:11.544155 10197 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:34:11.544245 10197 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:11.544327 10197 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:34:11.544370 10197 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:11.554407 10197 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.194:42987
I20260926 07:34:11.554448 10324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.194:42987 every 8 connection(s)
I20260926 07:34:11.555444 10197 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407783529717-22743-0/minicluster-data/ts-1/data/info.pb
I20260926 07:34:11.559448 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 10197
I20260926 07:34:11.559623 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407783529717-22743-0/minicluster-data/ts-1/wal/instance
I20260926 07:34:11.559959 22743 external_mini_cluster.cc:1428] Setting key 3a4b4d754883ec8856595ab3d0b83a4b
I20260926 07:34:11.562224 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407783529717-22743-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407783529717-22743-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407783529717-22743-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407783529717-22743-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.53.195:0
--local_ip_for_outbound_sockets=127.22.53.195
--webserver_interface=127.22.53.195
--webserver_port=0
--tserver_master_addrs=127.22.53.254:41839
--builtin_ntp_servers=127.22.53.212:37611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 07:34:11.565902 10045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:11.557620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.194:57179 (local address 127.22.53.254:41839)
0926 07:34:11.557798 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:11.557802 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:11.558929 (+ 1127us) server_negotiation.cc:407] Connection header received
0926 07:34:11.560497 (+ 1568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:11.560499 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:11.560617 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:11.560717 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:11.561937 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:11.562589 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:11.563769 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:11.564015 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:11.564131 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:11.565371 (+ 1240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:11.565375 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:11.565378 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:34:11.565392 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:11.565422 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:34:11.565430 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:11.565507 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:11.565679 (+ 172us) server_negotiation.cc:299] Negotiation successful
0926 07:34:11.565748 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260926 07:34:11.566823 10327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:11.558047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:41839 (local address 127.22.53.194:57179)
0926 07:34:11.558778 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:11.558819 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:11.560258 (+ 1439us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:11.560949 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:11.560959 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:11.561069 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:11.561776 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:11.561793 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:11.562726 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:11.562735 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:11.563638 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:11.563647 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:11.564179 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:11.564183 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:11.564361 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:11.565089 (+ 728us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:34:11.565090 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:11.565132 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:34:11.565205 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:34:11.565521 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:34:11.565531 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:34:11.565537 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 07:34:11.565670 (+ 133us) client_negotiation.cc:240] Negotiation successful
0926 07:34:11.565990 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":133,"threads_started":1}
I20260926 07:34:11.568215 10325 heartbeater.cc:347] Connected to a master server at 127.22.53.254:41839
I20260926 07:34:11.568470 10325 heartbeater.cc:464] Registering TS with master...
I20260926 07:34:11.569061 10325 heartbeater.cc:511] Master 127.22.53.254:41839 requested a full tablet report, sending...
I20260926 07:34:11.570245 10002 ts_manager.cc:194] Registered new tserver with Master: 435c5eed72a44782be096e57d3847ca0 (127.22.53.194:42987)
I20260926 07:34:11.570858 10002 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.53.194:57179
W20260926 07:34:11.682170 10328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:34:11.682528 10328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:34:11.682607 10328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:34:11.687023 10328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:34:11.687153 10328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:34:11.687258 10328 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.53.195
I20260926 07:34:11.692523 10328 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:37611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407783529717-22743-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407783529717-22743-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.53.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407783529717-22743-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.53.195
--webserver_port=0
--tserver_master_addrs=127.22.53.254:41839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10328
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.53.195
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407783529717-22743-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:34:11.693789 10328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:34:11.695243 10328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:34:11.705585 10333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:11.705578 10336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:11.705585 10334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:34:11.705668 10328 server_base.cc:1034] running on GCE node
I20260926 07:34:11.706284 10328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:34:11.706933 10328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:34:11.708098 10328 hybrid_clock.cc:648] HybridClock initialized: now 1790408051708080 us; error 39 us; skew 500 ppm
I20260926 07:34:11.710268 10328 webserver.cc:460] Webserver started at http://127.22.53.195:32877/ using document root <none> and password file <none>
I20260926 07:34:11.710896 10328 fs_manager.cc:359] Metadata directory not provided
I20260926 07:34:11.710966 10328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:34:11.711193 10328 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:34:11.713572 10328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407783529717-22743-0/minicluster-data/ts-2/data/instance:
uuid: "48c4c8ceab794aebb5d62688cb8259dd"
format_stamp: "Formatted at 2026-09-26 07:34:11 on dist-test-slave-wdjc"
server_key: "d7fbacc85b22becda622b6a8affdc11c"
server_key_iv: "5b6f35f357b069f0fea574b8461d55ef"
server_key_version: "encryptionkey@0"
I20260926 07:34:11.714208 10328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407783529717-22743-0/minicluster-data/ts-2/wal/instance:
uuid: "48c4c8ceab794aebb5d62688cb8259dd"
format_stamp: "Formatted at 2026-09-26 07:34:11 on dist-test-slave-wdjc"
server_key: "d7fbacc85b22becda622b6a8affdc11c"
server_key_iv: "5b6f35f357b069f0fea574b8461d55ef"
server_key_version: "encryptionkey@0"
I20260926 07:34:11.718707 10328 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 07:34:11.721452 10342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:11.722975 10328 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:34:11.723116 10328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407783529717-22743-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407783529717-22743-0/minicluster-data/ts-2/wal
uuid: "48c4c8ceab794aebb5d62688cb8259dd"
format_stamp: "Formatted at 2026-09-26 07:34:11 on dist-test-slave-wdjc"
server_key: "d7fbacc85b22becda622b6a8affdc11c"
server_key_iv: "5b6f35f357b069f0fea574b8461d55ef"
server_key_version: "encryptionkey@0"
I20260926 07:34:11.723248 10328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407783529717-22743-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407783529717-22743-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407783529717-22743-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:34:11.748859 10328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:34:11.749615 10328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:34:11.749832 10328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:34:11.750460 10328 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:34:11.751498 10328 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:34:11.751547 10328 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:11.751590 10328 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:34:11.751605 10328 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:11.761540 10328 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.195:33981
I20260926 07:34:11.761574 10455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.195:33981 every 8 connection(s)
I20260926 07:34:11.762527 10328 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407783529717-22743-0/minicluster-data/ts-2/data/info.pb
I20260926 07:34:11.769810 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 10328
I20260926 07:34:11.769920 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407783529717-22743-0/minicluster-data/ts-2/wal/instance
I20260926 07:34:11.770246 22743 external_mini_cluster.cc:1428] Setting key fdd186e2710894e78c089c8285d7eb36
I20260926 07:34:11.772572 10045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:11.764500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.195:39805 (local address 127.22.53.254:41839)
0926 07:34:11.764669 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:11.764674 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:11.765749 (+ 1075us) server_negotiation.cc:407] Connection header received
0926 07:34:11.767213 (+ 1464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:11.767216 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:11.767280 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:11.767388 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:11.768365 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:11.769004 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:11.770323 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:11.770559 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:11.770679 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:11.771992 (+ 1313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:11.771996 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:11.771999 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:34:11.772012 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:11.772042 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:34:11.772050 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:11.772127 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:11.772333 (+ 206us) server_negotiation.cc:299] Negotiation successful
0926 07:34:11.772401 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 07:34:11.773510 10458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:11.764878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:41839 (local address 127.22.53.195:39805)
0926 07:34:11.765575 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:11.765618 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:11.766969 (+ 1351us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:11.767581 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:11.767589 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:11.767680 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:11.768195 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:11.768207 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:11.769150 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:11.769154 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:11.770187 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:11.770197 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:11.770733 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:11.770737 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:11.770909 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:11.771695 (+ 786us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:34:11.771699 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:11.771760 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:34:11.771833 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:34:11.772184 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:34:11.772192 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:34:11.772197 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 07:34:11.772332 (+ 135us) client_negotiation.cc:240] Negotiation successful
0926 07:34:11.772671 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":112,"threads_started":1}
I20260926 07:34:11.774698 10456 heartbeater.cc:347] Connected to a master server at 127.22.53.254:41839
I20260926 07:34:11.774986 10456 heartbeater.cc:464] Registering TS with master...
I20260926 07:34:11.775573 10456 heartbeater.cc:511] Master 127.22.53.254:41839 requested a full tablet report, sending...
I20260926 07:34:11.776782 10002 ts_manager.cc:194] Registered new tserver with Master: 48c4c8ceab794aebb5d62688cb8259dd (127.22.53.195:33981)
I20260926 07:34:11.777397 10002 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.53.195:39805
I20260926 07:34:11.784804 22743 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:34:11.791764 10045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:11.787042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43218 (local address 127.22.53.254:41839)
0926 07:34:11.787224 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:11.787229 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:11.787378 (+ 149us) server_negotiation.cc:407] Connection header received
0926 07:34:11.787562 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:11.787565 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:11.787612 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:11.787706 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:11.788030 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:11.788633 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:11.789527 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:11.789749 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:11.789876 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:11.791250 (+ 1374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:11.791259 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:11.791262 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:34:11.791277 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:11.791302 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:34:11.791311 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:11.791400 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:11.791530 (+ 130us) server_negotiation.cc:299] Negotiation successful
0926 07:34:11.791598 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260926 07:34:11.794914 10002 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43218:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:34:11.798237 10002 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:34:11.814775 10467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:11.807886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.193:43629 (local address 127.0.0.1:59660)
0926 07:34:11.808790 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:11.808847 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:11.809501 (+ 654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:11.810119 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:11.810122 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:11.810138 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:34:11.810236 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:11.810243 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:11.811637 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:11.811642 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:11.813002 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:11.813014 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:11.814142 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:11.814157 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:11.814285 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:11.814350 (+ 65us) client_negotiation.cc:769] Sending connection context
0926 07:34:11.814440 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 07:34:11.814580 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":723,"mutex_wait_us":182,"spinlock_wait_cycles":4352,"thread_start_us":86,"threads_started":1}
I20260926 07:34:11.815013 10471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:11.809116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.195:33981 (local address 127.0.0.1:38718)
0926 07:34:11.809435 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:11.809450 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:11.809533 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:11.810420 (+ 887us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:11.810423 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:11.810440 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:34:11.810513 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:11.810519 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:11.811792 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:11.811798 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:11.813316 (+ 1518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:11.813328 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:11.814503 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:11.814506 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:11.814635 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:11.814673 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 07:34:11.814742 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 07:34:11.814822 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"mutex_wait_us":207,"thread_start_us":112,"threads_started":1}
I20260926 07:34:11.815240 10470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:11.808711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46066 (local address 127.22.53.194:42987)
0926 07:34:11.809130 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:11.809145 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:11.809182 (+ 37us) server_negotiation.cc:407] Connection header received
0926 07:34:11.809263 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:11.809269 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:11.809512 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:11.810101 (+ 589us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:34:11.810256 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:11.811542 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:11.813462 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:11.814246 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:11.814689 (+ 443us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:11.814875 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:11.814935 (+ 60us) server_negotiation.cc:299] Negotiation successful
0926 07:34:11.815048 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":111,"threads_started":1}
I20260926 07:34:11.815433 10469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:11.807985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.194:42987 (local address 127.0.0.1:46066)
0926 07:34:11.808815 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:11.808859 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:11.808991 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:11.809703 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:11.809709 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:11.809735 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:34:11.809900 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:11.809908 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:11.811679 (+ 1771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:11.811682 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:11.813316 (+ 1634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:11.813328 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:11.814380 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:11.814383 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:11.814472 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:11.814490 (+ 18us) client_negotiation.cc:769] Sending connection context
0926 07:34:11.814530 (+ 40us) client_negotiation.cc:240] Negotiation successful
0926 07:34:11.814603 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":752,"mutex_wait_us":216,"spinlock_wait_cycles":4992,"thread_start_us":130,"threads_started":1}
I20260926 07:34:11.815457 10472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:11.809580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38718 (local address 127.22.53.195:33981)
0926 07:34:11.809993 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:11.810012 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:11.810036 (+ 24us) server_negotiation.cc:407] Connection header received
0926 07:34:11.810119 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:11.810124 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:11.810304 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:11.810445 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:34:11.810609 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:11.811677 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:11.813485 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:11.814125 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:11.814233 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:11.814447 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:11.815163 (+ 716us) server_negotiation.cc:299] Negotiation successful
0926 07:34:11.815278 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":188,"threads_started":1}
I20260926 07:34:11.816136 10468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:11.807995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59660 (local address 127.22.53.193:43629)
0926 07:34:11.809272 (+ 1277us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:11.809279 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:11.809299 (+ 20us) server_negotiation.cc:407] Connection header received
0926 07:34:11.809743 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:11.809748 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:11.809943 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:11.810137 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:34:11.810333 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:11.811510 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:11.813140 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:11.813983 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:11.815642 (+ 1659us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:11.815842 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:11.815891 (+ 49us) server_negotiation.cc:299] Negotiation successful
0926 07:34:11.815986 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1176,"thread_start_us":81,"threads_started":1}
I20260926 07:34:11.818717 10390 tablet_service.cc:1467] Processing CreateTablet for tablet e53e6b9ab3d54224bc1cfff7c9433825 (DEFAULT_TABLE table=test-table [id=bdbe0a7cf2d24652b0638871afcef2fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:34:11.818914 10128 tablet_service.cc:1467] Processing CreateTablet for tablet e53e6b9ab3d54224bc1cfff7c9433825 (DEFAULT_TABLE table=test-table [id=bdbe0a7cf2d24652b0638871afcef2fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:34:11.818912 10259 tablet_service.cc:1467] Processing CreateTablet for tablet e53e6b9ab3d54224bc1cfff7c9433825 (DEFAULT_TABLE table=test-table [id=bdbe0a7cf2d24652b0638871afcef2fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:34:11.819835 10390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e53e6b9ab3d54224bc1cfff7c9433825. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:34:11.819940 10128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e53e6b9ab3d54224bc1cfff7c9433825. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:34:11.819940 10259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e53e6b9ab3d54224bc1cfff7c9433825. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:34:11.826807 10473 tablet_bootstrap.cc:492] T e53e6b9ab3d54224bc1cfff7c9433825 P 48c4c8ceab794aebb5d62688cb8259dd: Bootstrap starting.
I20260926 07:34:11.828274 10475 tablet_bootstrap.cc:492] T e53e6b9ab3d54224bc1cfff7c9433825 P 166722ce087b49c293c6a941db6cc467: Bootstrap starting.
I20260926 07:34:11.828893 10474 tablet_bootstrap.cc:492] T e53e6b9ab3d54224bc1cfff7c9433825 P 435c5eed72a44782be096e57d3847ca0: Bootstrap starting.
I20260926 07:34:11.829442 10473 tablet_bootstrap.cc:654] T e53e6b9ab3d54224bc1cfff7c9433825 P 48c4c8ceab794aebb5d62688cb8259dd: Neither blocks nor log segments found. Creating new log.
I20260926 07:34:11.830417 10473 log.cc:824] T e53e6b9ab3d54224bc1cfff7c9433825 P 48c4c8ceab794aebb5d62688cb8259dd: Log is configured to *not* fsync() on all Append() calls
I20260926 07:34:11.830976 10474 tablet_bootstrap.cc:654] T e53e6b9ab3d54224bc1cfff7c9433825 P 435c5eed72a44782be096e57d3847ca0: Neither blocks nor log segments found. Creating new log.
I20260926 07:34:11.831818 10474 log.cc:824] T e53e6b9ab3d54224bc1cfff7c9433825 P 435c5eed72a44782be096e57d3847ca0: Log is configured to *not* fsync() on all Append() calls
I20260926 07:34:11.832695 10473 tablet_bootstrap.cc:492] T e53e6b9ab3d54224bc1cfff7c9433825 P 48c4c8ceab794aebb5d62688cb8259dd: No bootstrap required, opened a new log
I20260926 07:34:11.832911 10473 ts_tablet_manager.cc:1397] T e53e6b9ab3d54224bc1cfff7c9433825 P 48c4c8ceab794aebb5d62688cb8259dd: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 07:34:11.833068 10475 tablet_bootstrap.cc:654] T e53e6b9ab3d54224bc1cfff7c9433825 P 166722ce087b49c293c6a941db6cc467: Neither blocks nor log segments found. Creating new log.
I20260926 07:34:11.833770 10475 log.cc:824] T e53e6b9ab3d54224bc1cfff7c9433825 P 166722ce087b49c293c6a941db6cc467: Log is configured to *not* fsync() on all Append() calls
I20260926 07:34:11.833967 10474 tablet_bootstrap.cc:492] T e53e6b9ab3d54224bc1cfff7c9433825 P 435c5eed72a44782be096e57d3847ca0: No bootstrap required, opened a new log
I20260926 07:34:11.834163 10474 ts_tablet_manager.cc:1397] T e53e6b9ab3d54224bc1cfff7c9433825 P 435c5eed72a44782be096e57d3847ca0: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 07:34:11.835801 10475 tablet_bootstrap.cc:492] T e53e6b9ab3d54224bc1cfff7c9433825 P 166722ce087b49c293c6a941db6cc467: No bootstrap required, opened a new log
I20260926 07:34:11.835989 10475 ts_tablet_manager.cc:1397] T e53e6b9ab3d54224bc1cfff7c9433825 P 166722ce087b49c293c6a941db6cc467: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260926 07:34:11.837832 10473 raft_consensus.cc:348] T e53e6b9ab3d54224bc1cfff7c9433825 P 48c4c8ceab794aebb5d62688cb8259dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "166722ce087b49c293c6a941db6cc467" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 43629 } } peers { permanent_uuid: "435c5eed72a44782be096e57d3847ca0" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 42987 } } peers { permanent_uuid: "48c4c8ceab794aebb5d62688cb8259dd" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 33981 } }
I20260926 07:34:11.838157 10473 raft_consensus.cc:374] T e53e6b9ab3d54224bc1cfff7c9433825 P 48c4c8ceab794aebb5d62688cb8259dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:34:11.838222 10473 raft_consensus.cc:729] T e53e6b9ab3d54224bc1cfff7c9433825 P 48c4c8ceab794aebb5d62688cb8259dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48c4c8ceab794aebb5d62688cb8259dd, State: Initialized, Role: FOLLOWER
I20260926 07:34:11.838511 10474 raft_consensus.cc:348] T e53e6b9ab3d54224bc1cfff7c9433825 P 435c5eed72a44782be096e57d3847ca0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "166722ce087b49c293c6a941db6cc467" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 43629 } } peers { permanent_uuid: "435c5eed72a44782be096e57d3847ca0" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 42987 } } peers { permanent_uuid: "48c4c8ceab794aebb5d62688cb8259dd" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 33981 } }
I20260926 07:34:11.838764 10474 raft_consensus.cc:374] T e53e6b9ab3d54224bc1cfff7c9433825 P 435c5eed72a44782be096e57d3847ca0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:34:11.838707 10473 consensus_queue.cc:260] T e53e6b9ab3d54224bc1cfff7c9433825 P 48c4c8ceab794aebb5d62688cb8259dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "166722ce087b49c293c6a941db6cc467" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 43629 } } peers { permanent_uuid: "435c5eed72a44782be096e57d3847ca0" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 42987 } } peers { permanent_uuid: "48c4c8ceab794aebb5d62688cb8259dd" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 33981 } }
I20260926 07:34:11.838816 10474 raft_consensus.cc:729] T e53e6b9ab3d54224bc1cfff7c9433825 P 435c5eed72a44782be096e57d3847ca0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 435c5eed72a44782be096e57d3847ca0, State: Initialized, Role: FOLLOWER
I20260926 07:34:11.839223 10474 consensus_queue.cc:260] T e53e6b9ab3d54224bc1cfff7c9433825 P 435c5eed72a44782be096e57d3847ca0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "166722ce087b49c293c6a941db6cc467" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 43629 } } peers { permanent_uuid: "435c5eed72a44782be096e57d3847ca0" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 42987 } } peers { permanent_uuid: "48c4c8ceab794aebb5d62688cb8259dd" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 33981 } }
I20260926 07:34:11.839828 10473 ts_tablet_manager.cc:1428] T e53e6b9ab3d54224bc1cfff7c9433825 P 48c4c8ceab794aebb5d62688cb8259dd: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260926 07:34:11.839743 10475 raft_consensus.cc:348] T e53e6b9ab3d54224bc1cfff7c9433825 P 166722ce087b49c293c6a941db6cc467 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "166722ce087b49c293c6a941db6cc467" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 43629 } } peers { permanent_uuid: "435c5eed72a44782be096e57d3847ca0" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 42987 } } peers { permanent_uuid: "48c4c8ceab794aebb5d62688cb8259dd" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 33981 } }
I20260926 07:34:11.839948 10325 heartbeater.cc:503] Master 127.22.53.254:41839 was elected leader, sending a full tablet report...
I20260926 07:34:11.839977 10475 raft_consensus.cc:374] T e53e6b9ab3d54224bc1cfff7c9433825 P 166722ce087b49c293c6a941db6cc467 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:34:11.840022 10475 raft_consensus.cc:729] T e53e6b9ab3d54224bc1cfff7c9433825 P 166722ce087b49c293c6a941db6cc467 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 166722ce087b49c293c6a941db6cc467, State: Initialized, Role: FOLLOWER
I20260926 07:34:11.840137 10474 ts_tablet_manager.cc:1428] T e53e6b9ab3d54224bc1cfff7c9433825 P 435c5eed72a44782be096e57d3847ca0: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260926 07:34:11.840399 10475 consensus_queue.cc:260] T e53e6b9ab3d54224bc1cfff7c9433825 P 166722ce087b49c293c6a941db6cc467 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "166722ce087b49c293c6a941db6cc467" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 43629 } } peers { permanent_uuid: "435c5eed72a44782be096e57d3847ca0" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 42987 } } peers { permanent_uuid: "48c4c8ceab794aebb5d62688cb8259dd" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 33981 } }
I20260926 07:34:11.840770 10456 heartbeater.cc:503] Master 127.22.53.254:41839 was elected leader, sending a full tablet report...
I20260926 07:34:11.841074 10194 heartbeater.cc:503] Master 127.22.53.254:41839 was elected leader, sending a full tablet report...
I20260926 07:34:11.841354 10475 ts_tablet_manager.cc:1428] T e53e6b9ab3d54224bc1cfff7c9433825 P 166722ce087b49c293c6a941db6cc467: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
W20260926 07:34:11.842267 10195 tablet.cc:2369] T e53e6b9ab3d54224bc1cfff7c9433825 P 166722ce087b49c293c6a941db6cc467: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:34:11.996050 10481 raft_consensus.cc:482] T e53e6b9ab3d54224bc1cfff7c9433825 P 166722ce087b49c293c6a941db6cc467 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:34:11.996363 10481 raft_consensus.cc:504] T e53e6b9ab3d54224bc1cfff7c9433825 P 166722ce087b49c293c6a941db6cc467 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "166722ce087b49c293c6a941db6cc467" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 43629 } } peers { permanent_uuid: "435c5eed72a44782be096e57d3847ca0" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 42987 } } peers { permanent_uuid: "48c4c8ceab794aebb5d62688cb8259dd" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 33981 } }
I20260926 07:34:11.997689 10481 leader_election.cc:288] T e53e6b9ab3d54224bc1cfff7c9433825 P 166722ce087b49c293c6a941db6cc467 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 435c5eed72a44782be096e57d3847ca0 (127.22.53.194:42987), 48c4c8ceab794aebb5d62688cb8259dd (127.22.53.195:33981)
I20260926 07:34:12.001904 10483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:11.997933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.195:33981 (local address 127.22.53.193:43011)
0926 07:34:11.998378 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:11.998393 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:11.998502 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:11.998781 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:11.998783 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:11.998798 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:34:11.998926 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:11.998932 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:11.999686 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:11.999689 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:12.000580 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:12.000590 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:12.001398 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:12.001405 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:12.001507 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:12.001530 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 07:34:12.001611 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 07:34:12.001688 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"mutex_wait_us":35,"thread_start_us":124,"threads_started":1}
I20260926 07:34:12.001919 10472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:11.998046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.193:43011 (local address 127.22.53.195:33981)
0926 07:34:11.998237 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:11.998241 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:11.998496 (+ 255us) server_negotiation.cc:407] Connection header received
0926 07:34:11.998590 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:11.998613 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:11.998674 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:11.998750 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:34:11.999043 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:11.999527 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:12.000722 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:12.001270 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:12.001377 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:12.001418 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:12.001639 (+ 221us) server_negotiation.cc:299] Negotiation successful
0926 07:34:12.001739 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260926 07:34:12.002017 10470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:11.997985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.53.193:57497 (local address 127.22.53.194:42987)
0926 07:34:11.998126 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:11.998130 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:11.998381 (+ 251us) server_negotiation.cc:407] Connection header received
0926 07:34:11.998485 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:11.998488 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:11.998546 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:11.998632 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:34:11.998962 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:11.999502 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:12.000939 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:12.001373 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:12.001494 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:12.001541 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:12.001770 (+ 229us) server_negotiation.cc:299] Negotiation successful
0926 07:34:12.001877 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 07:34:12.002532 10482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:11.997918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.194:42987 (local address 127.22.53.193:57497)
0926 07:34:11.998260 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:11.998277 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:11.998389 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:11.998653 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:11.998656 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:11.998672 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:34:11.998832 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:11.998839 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:11.999624 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:11.999630 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:12.000633 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:12.000814 (+ 181us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:12.001509 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:12.001512 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:12.001624 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:12.001660 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 07:34:12.001735 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 07:34:12.001867 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"mutex_wait_us":43,"thread_start_us":91,"threads_started":1}
I20260926 07:34:12.002631 10410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e53e6b9ab3d54224bc1cfff7c9433825" candidate_uuid: "166722ce087b49c293c6a941db6cc467" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48c4c8ceab794aebb5d62688cb8259dd" is_pre_election: true
I20260926 07:34:12.003012 10410 raft_consensus.cc:2454] T e53e6b9ab3d54224bc1cfff7c9433825 P 48c4c8ceab794aebb5d62688cb8259dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 166722ce087b49c293c6a941db6cc467 in term 0.
I20260926 07:34:12.003371 10279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e53e6b9ab3d54224bc1cfff7c9433825" candidate_uuid: "166722ce087b49c293c6a941db6cc467" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "435c5eed72a44782be096e57d3847ca0" is_pre_election: true
I20260926 07:34:12.003535 10083 leader_election.cc:302] T e53e6b9ab3d54224bc1cfff7c9433825 P 166722ce087b49c293c6a941db6cc467 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 166722ce087b49c293c6a941db6cc467, 48c4c8ceab794aebb5d62688cb8259dd; no voters:
I20260926 07:34:12.003659 10279 raft_consensus.cc:2454] T e53e6b9ab3d54224bc1cfff7c9433825 P 435c5eed72a44782be096e57d3847ca0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 166722ce087b49c293c6a941db6cc467 in term 0.
I20260926 07:34:12.003836 10481 raft_consensus.cc:2781] T e53e6b9ab3d54224bc1cfff7c9433825 P 166722ce087b49c293c6a941db6cc467 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:34:12.003937 10481 raft_consensus.cc:482] T e53e6b9ab3d54224bc1cfff7c9433825 P 166722ce087b49c293c6a941db6cc467 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:34:12.004021 10481 raft_consensus.cc:3037] T e53e6b9ab3d54224bc1cfff7c9433825 P 166722ce087b49c293c6a941db6cc467 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:34:12.005198 10481 raft_consensus.cc:504] T e53e6b9ab3d54224bc1cfff7c9433825 P 166722ce087b49c293c6a941db6cc467 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "166722ce087b49c293c6a941db6cc467" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 43629 } } peers { permanent_uuid: "435c5eed72a44782be096e57d3847ca0" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 42987 } } peers { permanent_uuid: "48c4c8ceab794aebb5d62688cb8259dd" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 33981 } }
I20260926 07:34:12.005563 10481 leader_election.cc:288] T e53e6b9ab3d54224bc1cfff7c9433825 P 166722ce087b49c293c6a941db6cc467 [CANDIDATE]: Term 1 election: Requested vote from peers 435c5eed72a44782be096e57d3847ca0 (127.22.53.194:42987), 48c4c8ceab794aebb5d62688cb8259dd (127.22.53.195:33981)
I20260926 07:34:12.005920 10279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e53e6b9ab3d54224bc1cfff7c9433825" candidate_uuid: "166722ce087b49c293c6a941db6cc467" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "435c5eed72a44782be096e57d3847ca0"
I20260926 07:34:12.006004 10410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e53e6b9ab3d54224bc1cfff7c9433825" candidate_uuid: "166722ce087b49c293c6a941db6cc467" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48c4c8ceab794aebb5d62688cb8259dd"
I20260926 07:34:12.006070 10279 raft_consensus.cc:3037] T e53e6b9ab3d54224bc1cfff7c9433825 P 435c5eed72a44782be096e57d3847ca0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:34:12.006177 10410 raft_consensus.cc:3037] T e53e6b9ab3d54224bc1cfff7c9433825 P 48c4c8ceab794aebb5d62688cb8259dd [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:34:12.007153 10279 raft_consensus.cc:2454] T e53e6b9ab3d54224bc1cfff7c9433825 P 435c5eed72a44782be096e57d3847ca0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 166722ce087b49c293c6a941db6cc467 in term 1.
I20260926 07:34:12.007333 10410 raft_consensus.cc:2454] T e53e6b9ab3d54224bc1cfff7c9433825 P 48c4c8ceab794aebb5d62688cb8259dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 166722ce087b49c293c6a941db6cc467 in term 1.
I20260926 07:34:12.007515 10081 leader_election.cc:302] T e53e6b9ab3d54224bc1cfff7c9433825 P 166722ce087b49c293c6a941db6cc467 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 166722ce087b49c293c6a941db6cc467, 435c5eed72a44782be096e57d3847ca0; no voters:
I20260926 07:34:12.007709 10481 raft_consensus.cc:2781] T e53e6b9ab3d54224bc1cfff7c9433825 P 166722ce087b49c293c6a941db6cc467 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:34:12.007975 10481 raft_consensus.cc:686] T e53e6b9ab3d54224bc1cfff7c9433825 P 166722ce087b49c293c6a941db6cc467 [term 1 LEADER]: Becoming Leader. State: Replica: 166722ce087b49c293c6a941db6cc467, State: Running, Role: LEADER
I20260926 07:34:12.008387 10481 consensus_queue.cc:237] T e53e6b9ab3d54224bc1cfff7c9433825 P 166722ce087b49c293c6a941db6cc467 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "166722ce087b49c293c6a941db6cc467" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 43629 } } peers { permanent_uuid: "435c5eed72a44782be096e57d3847ca0" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 42987 } } peers { permanent_uuid: "48c4c8ceab794aebb5d62688cb8259dd" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 33981 } }
I20260926 07:34:12.011766 10005 catalog_manager.cc:5495] T e53e6b9ab3d54224bc1cfff7c9433825 P 166722ce087b49c293c6a941db6cc467 reported cstate change: term changed from 0 to 1, leader changed from <none> to 166722ce087b49c293c6a941db6cc467 (127.22.53.193). New cstate: current_term: 1 leader_uuid: "166722ce087b49c293c6a941db6cc467" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "166722ce087b49c293c6a941db6cc467" member_type: VOTER last_known_addr { host: "127.22.53.193" port: 43629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "435c5eed72a44782be096e57d3847ca0" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 42987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48c4c8ceab794aebb5d62688cb8259dd" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 33981 } health_report { overall_health: UNKNOWN } } }
W20260926 07:34:12.013561 10457 tablet.cc:2369] T e53e6b9ab3d54224bc1cfff7c9433825 P 48c4c8ceab794aebb5d62688cb8259dd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 07:34:12.056694 10326 tablet.cc:2369] T e53e6b9ab3d54224bc1cfff7c9433825 P 435c5eed72a44782be096e57d3847ca0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:34:12.058097 10468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:12.054474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59664 (local address 127.22.53.193:43629)
0926 07:34:12.054608 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:12.054612 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:12.054626 (+ 14us) server_negotiation.cc:407] Connection header received
0926 07:34:12.054710 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:12.054712 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:12.054769 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:12.054896 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:34:12.055164 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:12.055669 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:12.056573 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:12.056804 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:12.056924 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:12.057250 (+ 326us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:34:12.057677 (+ 427us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:34:12.057824 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:12.057887 (+ 63us) server_negotiation.cc:299] Negotiation successful
0926 07:34:12.057946 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 07:34:12.064239 10279 raft_consensus.cc:1261] T e53e6b9ab3d54224bc1cfff7c9433825 P 435c5eed72a44782be096e57d3847ca0 [term 1 FOLLOWER]: Refusing update from remote peer 166722ce087b49c293c6a941db6cc467: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:34:12.064239 10410 raft_consensus.cc:1261] T e53e6b9ab3d54224bc1cfff7c9433825 P 48c4c8ceab794aebb5d62688cb8259dd [term 1 FOLLOWER]: Refusing update from remote peer 166722ce087b49c293c6a941db6cc467: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:34:12.065099 10484 consensus_queue.cc:1035] T e53e6b9ab3d54224bc1cfff7c9433825 P 166722ce087b49c293c6a941db6cc467 [LEADER]: Connected to new peer: Peer: permanent_uuid: "435c5eed72a44782be096e57d3847ca0" member_type: VOTER last_known_addr { host: "127.22.53.194" port: 42987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:34:12.065382 10481 consensus_queue.cc:1035] T e53e6b9ab3d54224bc1cfff7c9433825 P 166722ce087b49c293c6a941db6cc467 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48c4c8ceab794aebb5d62688cb8259dd" member_type: VOTER last_known_addr { host: "127.22.53.195" port: 33981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:34:12.074761 10488 mvcc.cc:205] Tried to move back new op lower bound from 7333511381247459328 to 7333511381029502976. Current Snapshot: MvccSnapshot[applied={T|T < 7333511381247459328}]
I20260926 07:34:12.077425 10489 mvcc.cc:205] Tried to move back new op lower bound from 7333511381247459328 to 7333511381029502976. Current Snapshot: MvccSnapshot[applied={T|T < 7333511381247459328}]
I20260926 07:34:12.083122 10005 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43218:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:34:12.083308 10005 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43218:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:34:12.086272 10005 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 56c23671a80d4ad3a8a85913f6dca39e: Sending DeleteTablet for 3 replicas of tablet e53e6b9ab3d54224bc1cfff7c9433825
I20260926 07:34:12.087255 10259 tablet_service.cc:1514] Processing DeleteTablet for tablet e53e6b9ab3d54224bc1cfff7c9433825 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:34:12 UTC) from {username='slave'} at 127.0.0.1:46066
I20260926 07:34:12.087442 10128 tablet_service.cc:1514] Processing DeleteTablet for tablet e53e6b9ab3d54224bc1cfff7c9433825 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:34:12 UTC) from {username='slave'} at 127.0.0.1:59660
I20260926 07:34:12.087815 10390 tablet_service.cc:1514] Processing DeleteTablet for tablet e53e6b9ab3d54224bc1cfff7c9433825 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:34:12 UTC) from {username='slave'} at 127.0.0.1:38718
I20260926 07:34:12.088148 10498 tablet_replica.cc:330] T e53e6b9ab3d54224bc1cfff7c9433825 P 166722ce087b49c293c6a941db6cc467: stopping tablet replica
I20260926 07:34:12.088590 10498 raft_consensus.cc:2229] T e53e6b9ab3d54224bc1cfff7c9433825 P 166722ce087b49c293c6a941db6cc467 [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:34:12.089058 10498 raft_consensus.cc:2258] T e53e6b9ab3d54224bc1cfff7c9433825 P 166722ce087b49c293c6a941db6cc467 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:34:12.089074 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 10066
I20260926 07:34:12.091121 10497 tablet_replica.cc:330] T e53e6b9ab3d54224bc1cfff7c9433825 P 435c5eed72a44782be096e57d3847ca0: stopping tablet replica
I20260926 07:34:12.091490 10497 raft_consensus.cc:2229] T e53e6b9ab3d54224bc1cfff7c9433825 P 435c5eed72a44782be096e57d3847ca0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:34:12.091845 10497 raft_consensus.cc:2258] T e53e6b9ab3d54224bc1cfff7c9433825 P 435c5eed72a44782be096e57d3847ca0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:34:12.092238 10499 tablet_replica.cc:330] T e53e6b9ab3d54224bc1cfff7c9433825 P 48c4c8ceab794aebb5d62688cb8259dd: stopping tablet replica
I20260926 07:34:12.092559 10499 raft_consensus.cc:2229] T e53e6b9ab3d54224bc1cfff7c9433825 P 48c4c8ceab794aebb5d62688cb8259dd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:34:12.092949 10499 raft_consensus.cc:2258] T e53e6b9ab3d54224bc1cfff7c9433825 P 48c4c8ceab794aebb5d62688cb8259dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:34:12.093892 10497 ts_tablet_manager.cc:1905] T e53e6b9ab3d54224bc1cfff7c9433825 P 435c5eed72a44782be096e57d3847ca0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:34:12.094362 10499 ts_tablet_manager.cc:1905] T e53e6b9ab3d54224bc1cfff7c9433825 P 48c4c8ceab794aebb5d62688cb8259dd: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 07:34:12.096576 9992 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.53.193:43629 (error 108)
I20260926 07:34:12.097234 10497 ts_tablet_manager.cc:1918] T e53e6b9ab3d54224bc1cfff7c9433825 P 435c5eed72a44782be096e57d3847ca0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:34:12.097259 10499 ts_tablet_manager.cc:1918] T e53e6b9ab3d54224bc1cfff7c9433825 P 48c4c8ceab794aebb5d62688cb8259dd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:34:12.097340 10497 log.cc:1196] T e53e6b9ab3d54224bc1cfff7c9433825 P 435c5eed72a44782be096e57d3847ca0: Deleting WAL directory at /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407783529717-22743-0/minicluster-data/ts-1/wal/wals/e53e6b9ab3d54224bc1cfff7c9433825
I20260926 07:34:12.097487 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 10197
I20260926 07:34:12.097801 10469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:12.097399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.193:43629 (local address 127.0.0.1:59666)
0926 07:34:12.097607 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:12.097698 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.193:43629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260926 07:34:12.097354 10499 log.cc:1196] T e53e6b9ab3d54224bc1cfff7c9433825 P 48c4c8ceab794aebb5d62688cb8259dd: Deleting WAL directory at /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407783529717-22743-0/minicluster-data/ts-2/wal/wals/e53e6b9ab3d54224bc1cfff7c9433825
W20260926 07:34:12.098806 9992 catalog_manager.cc:4584] TS 166722ce087b49c293c6a941db6cc467 (127.22.53.193:43629): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e53e6b9ab3d54224bc1cfff7c9433825: Network error: Client connection negotiation failed: client connection to 127.22.53.193:43629: connect: Connection refused (error 111)
I20260926 07:34:12.098985 10499 ts_tablet_manager.cc:1939] T e53e6b9ab3d54224bc1cfff7c9433825 P 48c4c8ceab794aebb5d62688cb8259dd: Deleting consensus metadata
I20260926 07:34:12.100287 9992 catalog_manager.cc:4841] TS 48c4c8ceab794aebb5d62688cb8259dd (127.22.53.195:33981): tablet e53e6b9ab3d54224bc1cfff7c9433825 (table test-table [id=bdbe0a7cf2d24652b0638871afcef2fa]) successfully deleted
I20260926 07:34:12.104106 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 10328
I20260926 07:34:12.104974 10469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:12.104285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.194:42987 (local address 127.0.0.1:46074)
0926 07:34:12.104791 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:12.104860 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.194:42987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403}
I20260926 07:34:12.110177 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 9975
2026-09-26T07:34:12Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1246 ms)
[----------] 6 tests from EncryptionPolicyTest (11029 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407783529717-22743-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:34:12 dist-test-slave-wdjc krb5kdc[10503](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:34:12 dist-test-slave-wdjc krb5kdc[10503](info): set up 2 sockets
Sep 26 07:34:12 dist-test-slave-wdjc krb5kdc[10503](info): commencing operation
krb5kdc: starting...
W20260926 07:34:13.152983 22743 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407783529717-22743-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407783529717-22743-0/krb5kdc/test-user.keytab.
Sep 26 07:34:13 dist-test-slave-wdjc krb5kdc[10503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408053, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:34:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:34:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407783529717-22743-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407783529717-22743-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 07:34:13.283644 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407783529717-22743-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407783529717-22743-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407783529717-22743-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45239
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407783529717-22743-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45239
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:34:13.404251 10516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:34:13.404681 10516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:34:13.404778 10516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:34:13.409189 10516 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:34:13.409324 10516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:34:13.409364 10516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:34:13.409432 10516 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:34:13.409471 10516 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:34:13.415371 10516 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407783529717-22743-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407783529717-22743-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45239
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407783529717-22743-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10516
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407783529717-22743-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:34:13.416688 10516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:34:13.417821 10516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:34:13.427883 10522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:13.427887 10524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:34:13.428004 10516 server_base.cc:1034] running on GCE node
W20260926 07:34:13.427887 10521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:34:13.429037 10516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:34:13.430361 10516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:34:13.431645 10516 hybrid_clock.cc:648] HybridClock initialized: now 1790408053431621 us; error 37 us; skew 500 ppm
Sep 26 07:34:13 dist-test-slave-wdjc krb5kdc[10503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408053, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:34:13.440172 10516 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 07:34:13.441433 10516 webserver.cc:460] Webserver started at http://127.0.0.1:44595/ using document root <none> and password file <none>
I20260926 07:34:13.442085 10516 fs_manager.cc:359] Metadata directory not provided
I20260926 07:34:13.442186 10516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:34:13.442448 10516 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:34:13.444810 10516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407783529717-22743-0/minicluster-data/master-0/data/instance:
uuid: "3d7d42158aec4e8988877597b2e72c8f"
format_stamp: "Formatted at 2026-09-26 07:34:13 on dist-test-slave-wdjc"
server_key: "a9383a62a094347fc7de4af5a23e7ad2"
server_key_iv: "3217f9db8fb464eebfa7275973e61b51"
server_key_version: "encryptionkey@0"
I20260926 07:34:13.445439 10516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407783529717-22743-0/minicluster-data/master-0/wal/instance:
uuid: "3d7d42158aec4e8988877597b2e72c8f"
format_stamp: "Formatted at 2026-09-26 07:34:13 on dist-test-slave-wdjc"
server_key: "a9383a62a094347fc7de4af5a23e7ad2"
server_key_iv: "3217f9db8fb464eebfa7275973e61b51"
server_key_version: "encryptionkey@0"
I20260926 07:34:13.449965 10516 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260926 07:34:13.453310 10531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:13.455264 10516 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260926 07:34:13.455525 10516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407783529717-22743-0/minicluster-data/master-0/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407783529717-22743-0/minicluster-data/master-0/wal
uuid: "3d7d42158aec4e8988877597b2e72c8f"
format_stamp: "Formatted at 2026-09-26 07:34:13 on dist-test-slave-wdjc"
server_key: "a9383a62a094347fc7de4af5a23e7ad2"
server_key_iv: "3217f9db8fb464eebfa7275973e61b51"
server_key_version: "encryptionkey@0"
I20260926 07:34:13.455863 10516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407783529717-22743-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407783529717-22743-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407783529717-22743-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:34:13.485268 10516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:34:13.488871 10516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:34:13.489063 10516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:34:13.499697 10516 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45239
I20260926 07:34:13.499771 10583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45239 every 8 connection(s)
I20260926 07:34:13.501027 10516 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
I20260926 07:34:13.501926 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 10516
I20260926 07:34:13.502080 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407783529717-22743-0/minicluster-data/master-0/wal/instance
I20260926 07:34:13.502427 22743 external_mini_cluster.cc:1428] Setting key 831210488abe1e55edf460df881450f8
I20260926 07:34:13.504922 10584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:34:13.511513 10584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d7d42158aec4e8988877597b2e72c8f: Bootstrap starting.
Sep 26 07:34:13 dist-test-slave-wdjc krb5kdc[10503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408053, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 07:34:13.514712 10584 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d7d42158aec4e8988877597b2e72c8f: Neither blocks nor log segments found. Creating new log.
I20260926 07:34:13.515764 10584 log.cc:824] T 00000000000000000000000000000000 P 3d7d42158aec4e8988877597b2e72c8f: Log is configured to *not* fsync() on all Append() calls
I20260926 07:34:13.518672 10584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d7d42158aec4e8988877597b2e72c8f: No bootstrap required, opened a new log
I20260926 07:34:13.520097 10587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:13.504116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44781 (local address 127.0.0.1:45239)
0926 07:34:13.504738 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:13.504755 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:13.504796 (+ 41us) server_negotiation.cc:407] Connection header received
0926 07:34:13.505635 (+ 839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:13.505664 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:13.506063 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:13.506462 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:13.507406 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:13.508503 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:13.509534 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:13.510061 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:13.510251 (+ 190us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:13.513410 (+ 3159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:13.513418 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:13.513435 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:13.513490 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:13.515788 (+ 2298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:13.516586 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:13.516594 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:13.516601 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:13.516774 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:13.517224 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:13.517227 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:13.517228 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:13.517713 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:13.517881 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:13.518385 (+ 504us) server_negotiation.cc:299] Negotiation successful
0926 07:34:13.518973 (+ 588us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":158,"threads_started":1}
I20260926 07:34:13.523283 10584 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d7d42158aec4e8988877597b2e72c8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d7d42158aec4e8988877597b2e72c8f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45239 } }
I20260926 07:34:13.523710 10584 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3d7d42158aec4e8988877597b2e72c8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:34:13.523819 10584 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d7d42158aec4e8988877597b2e72c8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d7d42158aec4e8988877597b2e72c8f, State: Initialized, Role: FOLLOWER
I20260926 07:34:13.524431 10584 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d7d42158aec4e8988877597b2e72c8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d7d42158aec4e8988877597b2e72c8f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45239 } }
I20260926 07:34:13.524852 10584 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3d7d42158aec4e8988877597b2e72c8f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:34:13.524988 10584 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3d7d42158aec4e8988877597b2e72c8f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:34:13.525141 10584 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d7d42158aec4e8988877597b2e72c8f [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:34:13.525632 22743 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:34:13.527245 10584 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3d7d42158aec4e8988877597b2e72c8f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d7d42158aec4e8988877597b2e72c8f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45239 } }
I20260926 07:34:13.527832 10584 leader_election.cc:302] T 00000000000000000000000000000000 P 3d7d42158aec4e8988877597b2e72c8f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3d7d42158aec4e8988877597b2e72c8f; no voters:
I20260926 07:34:13.528296 10584 leader_election.cc:288] T 00000000000000000000000000000000 P 3d7d42158aec4e8988877597b2e72c8f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:34:13.528427 10593 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3d7d42158aec4e8988877597b2e72c8f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:34:13.528723 10593 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3d7d42158aec4e8988877597b2e72c8f [term 1 LEADER]: Becoming Leader. State: Replica: 3d7d42158aec4e8988877597b2e72c8f, State: Running, Role: LEADER
I20260926 07:34:13.529136 10593 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3d7d42158aec4e8988877597b2e72c8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d7d42158aec4e8988877597b2e72c8f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45239 } }
I20260926 07:34:13.529958 10584 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d7d42158aec4e8988877597b2e72c8f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:34:13.530503 10593 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d7d42158aec4e8988877597b2e72c8f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d7d42158aec4e8988877597b2e72c8f. Latest consensus state: current_term: 1 leader_uuid: "3d7d42158aec4e8988877597b2e72c8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d7d42158aec4e8988877597b2e72c8f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45239 } } }
I20260926 07:34:13.530642 10593 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d7d42158aec4e8988877597b2e72c8f [sys.catalog]: This master's current role is: LEADER
I20260926 07:34:13.531038 10599 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:34:13.531143 10594 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d7d42158aec4e8988877597b2e72c8f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3d7d42158aec4e8988877597b2e72c8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d7d42158aec4e8988877597b2e72c8f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45239 } } }
I20260926 07:34:13.531356 10594 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d7d42158aec4e8988877597b2e72c8f [sys.catalog]: This master's current role is: LEADER
I20260926 07:34:13.534876 10599 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:34:13.542295 10599 catalog_manager.cc:1347] Generated new cluster ID: 83dfe0a17832400cb5a0892c0ed599e2
I20260926 07:34:13.542470 10599 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:34:13.543174 10587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:13.529035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39651 (local address 127.0.0.1:45239)
0926 07:34:13.529199 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:13.529204 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:13.532864 (+ 3660us) server_negotiation.cc:407] Connection header received
0926 07:34:13.533068 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:13.533072 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:13.533128 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:13.533183 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:13.534271 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:13.535054 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:13.536359 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:13.536753 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:13.536886 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:13.538991 (+ 2105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:13.538997 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:13.539001 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:13.539047 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:13.540946 (+ 1899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:13.541528 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:13.541532 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:13.541534 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:13.541624 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:13.542042 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:13.542045 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:13.542047 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:13.542328 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:13.542556 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:13.542910 (+ 354us) server_negotiation.cc:299] Negotiation successful
0926 07:34:13.543026 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 07:34:13.559577 10599 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:34:13.561133 10599 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:34:13.568605 10599 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3d7d42158aec4e8988877597b2e72c8f: Generated new TSK 0
I20260926 07:34:13.569612 10599 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:34:13.583451 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 10516
W20260926 07:34:13.589617 10501 connection.cc:535] client connection to 127.0.0.1:45239 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45239): Connection reset by peer (error 104)
2026-09-26T07:34:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1472 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407783529717-22743-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:34:13 dist-test-slave-wdjc krb5kdc[10615](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:34:13 dist-test-slave-wdjc krb5kdc[10615](info): set up 2 sockets
Sep 26 07:34:13 dist-test-slave-wdjc krb5kdc[10615](info): commencing operation
krb5kdc: starting...
W20260926 07:34:14.641016 22743 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407783529717-22743-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407783529717-22743-0/krb5kdc/test-user.keytab.
Sep 26 07:34:14 dist-test-slave-wdjc krb5kdc[10615](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408054, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:34:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:34:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407783529717-22743-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407783529717-22743-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 07:34:14.772693 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407783529717-22743-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407783529717-22743-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407783529717-22743-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46595
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407783529717-22743-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46595
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:34:14.890547 10628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:34:14.890898 10628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:34:14.890947 10628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:34:14.895315 10628 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:34:14.895411 10628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:34:14.895431 10628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:34:14.895452 10628 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:34:14.895468 10628 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:34:14.901445 10628 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407783529717-22743-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407783529717-22743-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46595
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407783529717-22743-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46595
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10628
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407783529717-22743-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:34:14.902705 10628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:34:14.903776 10628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:34:14.913004 10633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:14.913015 10636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:34:14.913290 10628 server_base.cc:1034] running on GCE node
W20260926 07:34:14.913065 10634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:34:14.914094 10628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:34:14.915314 10628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:34:14.916498 10628 hybrid_clock.cc:648] HybridClock initialized: now 1790408054916484 us; error 30 us; skew 500 ppm
Sep 26 07:34:14 dist-test-slave-wdjc krb5kdc[10615](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408054, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:34:14.919566 10628 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 07:34:14.920670 10628 webserver.cc:460] Webserver started at http://127.0.0.1:40611/ using document root <none> and password file <none>
I20260926 07:34:14.921277 10628 fs_manager.cc:359] Metadata directory not provided
I20260926 07:34:14.921372 10628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:34:14.921613 10628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:34:14.923866 10628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407783529717-22743-0/minicluster-data/master-0/data/instance:
uuid: "e0d2aabc2df4430c8d46afdfe072f5f1"
format_stamp: "Formatted at 2026-09-26 07:34:14 on dist-test-slave-wdjc"
server_key: "df8961dae345c65b93deaea537f796c1"
server_key_iv: "c8a5ff8cee803989c53eb94f16378ea4"
server_key_version: "encryptionkey@0"
I20260926 07:34:14.924481 10628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407783529717-22743-0/minicluster-data/master-0/wal/instance:
uuid: "e0d2aabc2df4430c8d46afdfe072f5f1"
format_stamp: "Formatted at 2026-09-26 07:34:14 on dist-test-slave-wdjc"
server_key: "df8961dae345c65b93deaea537f796c1"
server_key_iv: "c8a5ff8cee803989c53eb94f16378ea4"
server_key_version: "encryptionkey@0"
I20260926 07:34:14.928834 10628 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 07:34:14.931594 10643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:14.932971 10628 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:34:14.933127 10628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407783529717-22743-0/minicluster-data/master-0/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407783529717-22743-0/minicluster-data/master-0/wal
uuid: "e0d2aabc2df4430c8d46afdfe072f5f1"
format_stamp: "Formatted at 2026-09-26 07:34:14 on dist-test-slave-wdjc"
server_key: "df8961dae345c65b93deaea537f796c1"
server_key_iv: "c8a5ff8cee803989c53eb94f16378ea4"
server_key_version: "encryptionkey@0"
I20260926 07:34:14.933266 10628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407783529717-22743-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407783529717-22743-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407783529717-22743-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:34:14.956970 10628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:34:14.960755 10628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:34:14.960989 10628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:34:14.968962 10628 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46595
I20260926 07:34:14.969046 10695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46595 every 8 connection(s)
I20260926 07:34:14.970129 10628 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
I20260926 07:34:14.973214 10696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:34:14.979492 10696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0d2aabc2df4430c8d46afdfe072f5f1: Bootstrap starting.
I20260926 07:34:14.980237 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 10628
I20260926 07:34:14.980432 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407783529717-22743-0/minicluster-data/master-0/wal/instance
I20260926 07:34:14.980796 22743 external_mini_cluster.cc:1428] Setting key f5a34bf0c96fec71b9f4848f1dddbceb
I20260926 07:34:14.982600 10696 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e0d2aabc2df4430c8d46afdfe072f5f1: Neither blocks nor log segments found. Creating new log.
I20260926 07:34:14.983392 10696 log.cc:824] T 00000000000000000000000000000000 P e0d2aabc2df4430c8d46afdfe072f5f1: Log is configured to *not* fsync() on all Append() calls
I20260926 07:34:14.985707 10696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0d2aabc2df4430c8d46afdfe072f5f1: No bootstrap required, opened a new log
I20260926 07:34:14.990321 10696 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0d2aabc2df4430c8d46afdfe072f5f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0d2aabc2df4430c8d46afdfe072f5f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46595 } }
I20260926 07:34:14.990623 10696 raft_consensus.cc:374] T 00000000000000000000000000000000 P e0d2aabc2df4430c8d46afdfe072f5f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:34:14.990720 10696 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0d2aabc2df4430c8d46afdfe072f5f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0d2aabc2df4430c8d46afdfe072f5f1, State: Initialized, Role: FOLLOWER
I20260926 07:34:14.990863 10699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:14.982167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36635 (local address 127.0.0.1:46595)
0926 07:34:14.982619 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:14.982635 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:14.982671 (+ 36us) server_negotiation.cc:407] Connection header received
0926 07:34:14.983443 (+ 772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:14.983462 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:14.983951 (+ 489us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:14.984344 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:14.984768 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:14.985764 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:14.987001 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:14.987525 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:14.987674 (+ 149us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:14.988762 (+ 1088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:14.988768 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:14.988783 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:34:14.989240 (+ 457us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:14.989303 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:34:14.989329 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:14.989443 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:14.989750 (+ 307us) server_negotiation.cc:299] Negotiation successful
0926 07:34:14.989926 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":149,"threads_started":1}
I20260926 07:34:14.991225 10696 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0d2aabc2df4430c8d46afdfe072f5f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0d2aabc2df4430c8d46afdfe072f5f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46595 } }
I20260926 07:34:14.991382 10696 raft_consensus.cc:388] T 00000000000000000000000000000000 P e0d2aabc2df4430c8d46afdfe072f5f1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:34:14.991467 10696 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0d2aabc2df4430c8d46afdfe072f5f1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:34:14.991571 10696 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0d2aabc2df4430c8d46afdfe072f5f1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:34:14.992605 10696 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0d2aabc2df4430c8d46afdfe072f5f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0d2aabc2df4430c8d46afdfe072f5f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46595 } }
I20260926 07:34:14.993000 10696 leader_election.cc:302] T 00000000000000000000000000000000 P e0d2aabc2df4430c8d46afdfe072f5f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0d2aabc2df4430c8d46afdfe072f5f1; no voters:
I20260926 07:34:14.993374 10696 leader_election.cc:288] T 00000000000000000000000000000000 P e0d2aabc2df4430c8d46afdfe072f5f1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:34:14.993506 10701 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e0d2aabc2df4430c8d46afdfe072f5f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:34:14.993724 10701 raft_consensus.cc:686] T 00000000000000000000000000000000 P e0d2aabc2df4430c8d46afdfe072f5f1 [term 1 LEADER]: Becoming Leader. State: Replica: e0d2aabc2df4430c8d46afdfe072f5f1, State: Running, Role: LEADER
I20260926 07:34:14.994055 10701 consensus_queue.cc:237] T 00000000000000000000000000000000 P e0d2aabc2df4430c8d46afdfe072f5f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0d2aabc2df4430c8d46afdfe072f5f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46595 } }
I20260926 07:34:14.994274 22743 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:34:14.994789 10696 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0d2aabc2df4430c8d46afdfe072f5f1 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:34:14.995404 10701 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0d2aabc2df4430c8d46afdfe072f5f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0d2aabc2df4430c8d46afdfe072f5f1. Latest consensus state: current_term: 1 leader_uuid: "e0d2aabc2df4430c8d46afdfe072f5f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0d2aabc2df4430c8d46afdfe072f5f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46595 } } }
I20260926 07:34:14.995527 10701 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0d2aabc2df4430c8d46afdfe072f5f1 [sys.catalog]: This master's current role is: LEADER
I20260926 07:34:14.995844 10703 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0d2aabc2df4430c8d46afdfe072f5f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e0d2aabc2df4430c8d46afdfe072f5f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0d2aabc2df4430c8d46afdfe072f5f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46595 } } }
I20260926 07:34:14.995939 10703 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0d2aabc2df4430c8d46afdfe072f5f1 [sys.catalog]: This master's current role is: LEADER
I20260926 07:34:14.996243 10708 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:34:14.999770 10708 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:34:15.002481 10699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:14.996770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57755 (local address 127.0.0.1:46595)
0926 07:34:14.996881 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:14.996885 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:14.997530 (+ 645us) server_negotiation.cc:407] Connection header received
0926 07:34:14.997694 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:14.997698 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:14.997749 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:14.997801 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:14.998256 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:14.999014 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:15.000236 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:15.000596 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:15.000703 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:15.001899 (+ 1196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:15.001904 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:15.001909 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:34:15.001924 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:15.001955 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:34:15.001969 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:15.002055 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:15.002298 (+ 243us) server_negotiation.cc:299] Negotiation successful
0926 07:34:15.002340 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 07:34:15.006240 10708 catalog_manager.cc:1347] Generated new cluster ID: b2f89c6c2e66427dbb5ba0ad2cc0f3db
I20260926 07:34:15.006397 10708 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:34:15.027403 10708 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:34:15.028417 10708 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:34:15.034044 10708 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e0d2aabc2df4430c8d46afdfe072f5f1: Generated new TSK 0
I20260926 07:34:15.034875 10708 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:34:15.048811 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 10628
W20260926 07:34:15.054884 10613 connection.cc:535] client connection to 127.0.0.1:46595 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46595): Connection reset by peer (error 104)
2026-09-26T07:34:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1465 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407783529717-22743-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:34:15 dist-test-slave-wdjc krb5kdc[10727](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:34:15 dist-test-slave-wdjc krb5kdc[10727](info): set up 2 sockets
Sep 26 07:34:15 dist-test-slave-wdjc krb5kdc[10727](info): commencing operation
krb5kdc: starting...
W20260926 07:34:16.112977 22743 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.007s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407783529717-22743-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407783529717-22743-0/krb5kdc/test-user.keytab.
Sep 26 07:34:16 dist-test-slave-wdjc krb5kdc[10727](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408056, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:34:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:34:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407783529717-22743-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407783529717-22743-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 07:34:16.252923 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407783529717-22743-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407783529717-22743-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407783529717-22743-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45295
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407783529717-22743-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45295
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:34:16.389163 10740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:34:16.389596 10740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:34:16.389691 10740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:34:16.395046 10740 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:34:16.395202 10740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:34:16.395243 10740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:34:16.395289 10740 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:34:16.395365 10740 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:34:16.402817 10740 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407783529717-22743-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407783529717-22743-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45295
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407783529717-22743-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45295
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10740
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407783529717-22743-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:34:16.404125 10740 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:34:16.405314 10740 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:34:16.415802 10745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:16.415802 10746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:34:16.416227 10740 server_base.cc:1034] running on GCE node
W20260926 07:34:16.416096 10748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:34:16.417111 10740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:34:16.418462 10740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:34:16.419682 10740 hybrid_clock.cc:648] HybridClock initialized: now 1790408056419653 us; error 50 us; skew 500 ppm
Sep 26 07:34:16 dist-test-slave-wdjc krb5kdc[10727](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408056, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:34:16.423790 10740 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 07:34:16.424968 10740 webserver.cc:460] Webserver started at http://127.0.0.1:39565/ using document root <none> and password file <none>
I20260926 07:34:16.425655 10740 fs_manager.cc:359] Metadata directory not provided
I20260926 07:34:16.425760 10740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:34:16.426028 10740 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:34:16.428393 10740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407783529717-22743-0/minicluster-data/master-0/data/instance:
uuid: "a8d8669142aa4ab7a6bcbd17ecc56e55"
format_stamp: "Formatted at 2026-09-26 07:34:16 on dist-test-slave-wdjc"
server_key: "fecc075d7f3035552cfeb3d043cc1dfa"
server_key_iv: "3dbbf904987b8bc0ed5ffe9463370718"
server_key_version: "encryptionkey@0"
I20260926 07:34:16.429138 10740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407783529717-22743-0/minicluster-data/master-0/wal/instance:
uuid: "a8d8669142aa4ab7a6bcbd17ecc56e55"
format_stamp: "Formatted at 2026-09-26 07:34:16 on dist-test-slave-wdjc"
server_key: "fecc075d7f3035552cfeb3d043cc1dfa"
server_key_iv: "3dbbf904987b8bc0ed5ffe9463370718"
server_key_version: "encryptionkey@0"
I20260926 07:34:16.433660 10740 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 07:34:16.436712 10755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:16.438263 10740 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:34:16.438503 10740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407783529717-22743-0/minicluster-data/master-0/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407783529717-22743-0/minicluster-data/master-0/wal
uuid: "a8d8669142aa4ab7a6bcbd17ecc56e55"
format_stamp: "Formatted at 2026-09-26 07:34:16 on dist-test-slave-wdjc"
server_key: "fecc075d7f3035552cfeb3d043cc1dfa"
server_key_iv: "3dbbf904987b8bc0ed5ffe9463370718"
server_key_version: "encryptionkey@0"
I20260926 07:34:16.438679 10740 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407783529717-22743-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407783529717-22743-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407783529717-22743-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:34:16.445050 10740 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:34:16.449117 10740 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:34:16.449383 10740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:34:16.458269 10740 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45295
I20260926 07:34:16.458287 10807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45295 every 8 connection(s)
I20260926 07:34:16.459405 10740 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
I20260926 07:34:16.461591 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 10740
I20260926 07:34:16.461803 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407783529717-22743-0/minicluster-data/master-0/wal/instance
I20260926 07:34:16.462152 22743 external_mini_cluster.cc:1428] Setting key d4e62d77551a1f7f06d499fa69e637d0
I20260926 07:34:16.463023 10808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:34:16.470737 10811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:16.464300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47137 (local address 127.0.0.1:45295)
0926 07:34:16.464959 (+ 659us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:16.464978 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:16.465306 (+ 328us) server_negotiation.cc:407] Connection header received
0926 07:34:16.466070 (+ 764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:16.466090 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:16.466434 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:16.466781 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:16.468182 (+ 1401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:16.468190 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:16.468212 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:34:16.468274 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:16.468407 (+ 133us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:34:16.468498 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:16.468761 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:16.468866 (+ 105us) server_negotiation.cc:299] Negotiation successful
0926 07:34:16.469189 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":231,"threads_started":1}
I20260926 07:34:16.474279 10808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8d8669142aa4ab7a6bcbd17ecc56e55: Bootstrap starting.
I20260926 07:34:16.475575 22743 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:34:16.477303 10808 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a8d8669142aa4ab7a6bcbd17ecc56e55: Neither blocks nor log segments found. Creating new log.
I20260926 07:34:16.478229 10808 log.cc:824] T 00000000000000000000000000000000 P a8d8669142aa4ab7a6bcbd17ecc56e55: Log is configured to *not* fsync() on all Append() calls
I20260926 07:34:16.480472 10811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:16.478215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43777 (local address 127.0.0.1:45295)
0926 07:34:16.478390 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:16.478396 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:16.478637 (+ 241us) server_negotiation.cc:407] Connection header received
0926 07:34:16.478817 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:16.478821 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:16.478858 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:16.478969 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:16.479906 (+ 937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:16.479910 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:16.479912 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:34:16.479928 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:16.479957 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:34:16.479969 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:16.480061 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:16.480214 (+ 153us) server_negotiation.cc:299] Negotiation successful
0926 07:34:16.480255 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 07:34:16.481173 10808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8d8669142aa4ab7a6bcbd17ecc56e55: No bootstrap required, opened a new log
I20260926 07:34:16.485181 10808 raft_consensus.cc:348] T 00000000000000000000000000000000 P a8d8669142aa4ab7a6bcbd17ecc56e55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8d8669142aa4ab7a6bcbd17ecc56e55" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45295 } }
I20260926 07:34:16.485481 10808 raft_consensus.cc:374] T 00000000000000000000000000000000 P a8d8669142aa4ab7a6bcbd17ecc56e55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:34:16.485517 10808 raft_consensus.cc:729] T 00000000000000000000000000000000 P a8d8669142aa4ab7a6bcbd17ecc56e55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8d8669142aa4ab7a6bcbd17ecc56e55, State: Initialized, Role: FOLLOWER
I20260926 07:34:16.486053 10808 consensus_queue.cc:260] T 00000000000000000000000000000000 P a8d8669142aa4ab7a6bcbd17ecc56e55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8d8669142aa4ab7a6bcbd17ecc56e55" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45295 } }
I20260926 07:34:16.486202 10808 raft_consensus.cc:388] T 00000000000000000000000000000000 P a8d8669142aa4ab7a6bcbd17ecc56e55 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:34:16.486275 10808 raft_consensus.cc:482] T 00000000000000000000000000000000 P a8d8669142aa4ab7a6bcbd17ecc56e55 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:34:16.486550 10808 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a8d8669142aa4ab7a6bcbd17ecc56e55 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:34:16.488006 10808 raft_consensus.cc:504] T 00000000000000000000000000000000 P a8d8669142aa4ab7a6bcbd17ecc56e55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8d8669142aa4ab7a6bcbd17ecc56e55" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45295 } }
I20260926 07:34:16.488508 10808 leader_election.cc:302] T 00000000000000000000000000000000 P a8d8669142aa4ab7a6bcbd17ecc56e55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a8d8669142aa4ab7a6bcbd17ecc56e55; no voters:
I20260926 07:34:16.488979 10808 leader_election.cc:288] T 00000000000000000000000000000000 P a8d8669142aa4ab7a6bcbd17ecc56e55 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:34:16.489251 10818 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a8d8669142aa4ab7a6bcbd17ecc56e55 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:34:16.489578 10818 raft_consensus.cc:686] T 00000000000000000000000000000000 P a8d8669142aa4ab7a6bcbd17ecc56e55 [term 1 LEADER]: Becoming Leader. State: Replica: a8d8669142aa4ab7a6bcbd17ecc56e55, State: Running, Role: LEADER
I20260926 07:34:16.489982 10818 consensus_queue.cc:237] T 00000000000000000000000000000000 P a8d8669142aa4ab7a6bcbd17ecc56e55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8d8669142aa4ab7a6bcbd17ecc56e55" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45295 } }
I20260926 07:34:16.490535 10808 sys_catalog.cc:564] T 00000000000000000000000000000000 P a8d8669142aa4ab7a6bcbd17ecc56e55 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:34:16.491936 10819 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8d8669142aa4ab7a6bcbd17ecc56e55 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a8d8669142aa4ab7a6bcbd17ecc56e55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8d8669142aa4ab7a6bcbd17ecc56e55" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45295 } } }
I20260926 07:34:16.492115 10819 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8d8669142aa4ab7a6bcbd17ecc56e55 [sys.catalog]: This master's current role is: LEADER
I20260926 07:34:16.492413 10820 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8d8669142aa4ab7a6bcbd17ecc56e55 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a8d8669142aa4ab7a6bcbd17ecc56e55. Latest consensus state: current_term: 1 leader_uuid: "a8d8669142aa4ab7a6bcbd17ecc56e55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8d8669142aa4ab7a6bcbd17ecc56e55" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45295 } } }
I20260926 07:34:16.492643 10820 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8d8669142aa4ab7a6bcbd17ecc56e55 [sys.catalog]: This master's current role is: LEADER
I20260926 07:34:16.492739 10827 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:34:16.496197 10827 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:34:16.503470 10827 catalog_manager.cc:1347] Generated new cluster ID: 76a155af4e7645598f05de9435697d30
I20260926 07:34:16.503592 10827 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:34:16.517220 10827 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:34:16.517613 10827 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:34:16.525982 10827 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a8d8669142aa4ab7a6bcbd17ecc56e55: Generated new TSK 0
I20260926 07:34:16.526863 10827 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:34:16.531327 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 10740
2026-09-26T07:34:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1482 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407783529717-22743-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:34:16 dist-test-slave-wdjc krb5kdc[10839](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:34:16 dist-test-slave-wdjc krb5kdc[10839](info): set up 2 sockets
Sep 26 07:34:16 dist-test-slave-wdjc krb5kdc[10839](info): commencing operation
krb5kdc: starting...
W20260926 07:34:17.600979 22743 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407783529717-22743-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407783529717-22743-0/krb5kdc/test-user.keytab.
Sep 26 07:34:17 dist-test-slave-wdjc krb5kdc[10839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408057, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:34:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:34:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407783529717-22743-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407783529717-22743-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 07:34:17.727474 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407783529717-22743-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407783529717-22743-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407783529717-22743-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36819
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407783529717-22743-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36819
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:34:17.846462 10852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:34:17.846782 10852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:34:17.846827 10852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:34:17.851373 10852 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:34:17.851465 10852 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:34:17.851487 10852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:34:17.851505 10852 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:34:17.851521 10852 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:34:17.857545 10852 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407783529717-22743-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407783529717-22743-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36819
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407783529717-22743-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36819
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10852
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407783529717-22743-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:34:17.858793 10852 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:34:17.859933 10852 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:34:17.869298 10857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:17.869294 10860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:17.869294 10858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:34:17.869671 10852 server_base.cc:1034] running on GCE node
I20260926 07:34:17.870256 10852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:34:17.871492 10852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:34:17.872676 10852 hybrid_clock.cc:648] HybridClock initialized: now 1790408057872655 us; error 36 us; skew 500 ppm
Sep 26 07:34:17 dist-test-slave-wdjc krb5kdc[10839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408057, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:34:17.876344 10852 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 07:34:17.877455 10852 webserver.cc:460] Webserver started at http://127.0.0.1:40137/ using document root <none> and password file <none>
I20260926 07:34:17.878032 10852 fs_manager.cc:359] Metadata directory not provided
I20260926 07:34:17.878095 10852 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:34:17.878275 10852 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:34:17.880499 10852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407783529717-22743-0/minicluster-data/master-0/data/instance:
uuid: "5a2951c5b5924d13bdea862397966ad9"
format_stamp: "Formatted at 2026-09-26 07:34:17 on dist-test-slave-wdjc"
server_key: "9e48c3b668ab5848db2d768afa54fa27"
server_key_iv: "6ce8ea987267c6d1f46d52ac51d4e7bc"
server_key_version: "encryptionkey@0"
I20260926 07:34:17.881158 10852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407783529717-22743-0/minicluster-data/master-0/wal/instance:
uuid: "5a2951c5b5924d13bdea862397966ad9"
format_stamp: "Formatted at 2026-09-26 07:34:17 on dist-test-slave-wdjc"
server_key: "9e48c3b668ab5848db2d768afa54fa27"
server_key_iv: "6ce8ea987267c6d1f46d52ac51d4e7bc"
server_key_version: "encryptionkey@0"
I20260926 07:34:17.885252 10852 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 07:34:17.887915 10867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:17.889338 10852 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 07:34:17.889467 10852 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407783529717-22743-0/minicluster-data/master-0/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407783529717-22743-0/minicluster-data/master-0/wal
uuid: "5a2951c5b5924d13bdea862397966ad9"
format_stamp: "Formatted at 2026-09-26 07:34:17 on dist-test-slave-wdjc"
server_key: "9e48c3b668ab5848db2d768afa54fa27"
server_key_iv: "6ce8ea987267c6d1f46d52ac51d4e7bc"
server_key_version: "encryptionkey@0"
I20260926 07:34:17.889632 10852 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407783529717-22743-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407783529717-22743-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407783529717-22743-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:34:17.906179 10852 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:34:17.909832 10852 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:34:17.910079 10852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:34:17.918265 10852 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36819
I20260926 07:34:17.918291 10919 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36819 every 8 connection(s)
I20260926 07:34:17.919353 10852 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
I20260926 07:34:17.922395 10920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:34:17.925320 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 10852
I20260926 07:34:17.925462 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407783529717-22743-0/minicluster-data/master-0/wal/instance
I20260926 07:34:17.925806 22743 external_mini_cluster.cc:1428] Setting key b462e99c42817262f1075ca0d07ed00d
I20260926 07:34:17.929808 10920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a2951c5b5924d13bdea862397966ad9: Bootstrap starting.
I20260926 07:34:17.932801 10920 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a2951c5b5924d13bdea862397966ad9: Neither blocks nor log segments found. Creating new log.
I20260926 07:34:17.933725 10920 log.cc:824] T 00000000000000000000000000000000 P 5a2951c5b5924d13bdea862397966ad9: Log is configured to *not* fsync() on all Append() calls
I20260926 07:34:17.936187 10920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a2951c5b5924d13bdea862397966ad9: No bootstrap required, opened a new log
Sep 26 07:34:17 dist-test-slave-wdjc krb5kdc[10839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408057, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 07:34:17.939872 10920 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a2951c5b5924d13bdea862397966ad9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a2951c5b5924d13bdea862397966ad9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36819 } }
I20260926 07:34:17.940192 10920 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5a2951c5b5924d13bdea862397966ad9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:34:17.940294 10920 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a2951c5b5924d13bdea862397966ad9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a2951c5b5924d13bdea862397966ad9, State: Initialized, Role: FOLLOWER
I20260926 07:34:17.940866 10920 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a2951c5b5924d13bdea862397966ad9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a2951c5b5924d13bdea862397966ad9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36819 } }
I20260926 07:34:17.941033 10920 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5a2951c5b5924d13bdea862397966ad9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:34:17.941119 10920 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a2951c5b5924d13bdea862397966ad9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:34:17.941258 10920 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a2951c5b5924d13bdea862397966ad9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:34:17.942646 10920 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a2951c5b5924d13bdea862397966ad9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a2951c5b5924d13bdea862397966ad9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36819 } }
I20260926 07:34:17.943054 10920 leader_election.cc:302] T 00000000000000000000000000000000 P 5a2951c5b5924d13bdea862397966ad9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5a2951c5b5924d13bdea862397966ad9; no voters:
I20260926 07:34:17.943434 10920 leader_election.cc:288] T 00000000000000000000000000000000 P 5a2951c5b5924d13bdea862397966ad9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:34:17.943626 10925 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a2951c5b5924d13bdea862397966ad9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:34:17.943734 10923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:17.927440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35965 (local address 127.0.0.1:36819)
0926 07:34:17.927939 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:17.927959 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:17.928004 (+ 45us) server_negotiation.cc:407] Connection header received
0926 07:34:17.929012 (+ 1008us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:17.929038 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:17.929533 (+ 495us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:17.929938 (+ 405us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:17.930787 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:17.932333 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:17.933416 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:17.934147 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:17.934306 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:17.937474 (+ 3168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:17.937480 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:17.937496 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:17.937531 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:17.939979 (+ 2448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:17.940632 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:17.940638 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:17.940642 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:17.940774 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:17.941119 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:17.941125 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:17.941144 (+ 19us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:17.941759 (+ 615us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:17.942018 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:17.942252 (+ 234us) server_negotiation.cc:299] Negotiation successful
0926 07:34:17.942622 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":141,"threads_started":1}
I20260926 07:34:17.943871 10925 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5a2951c5b5924d13bdea862397966ad9 [term 1 LEADER]: Becoming Leader. State: Replica: 5a2951c5b5924d13bdea862397966ad9, State: Running, Role: LEADER
I20260926 07:34:17.944232 10925 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a2951c5b5924d13bdea862397966ad9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a2951c5b5924d13bdea862397966ad9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36819 } }
I20260926 07:34:17.944801 10920 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a2951c5b5924d13bdea862397966ad9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:34:17.946030 10926 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a2951c5b5924d13bdea862397966ad9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5a2951c5b5924d13bdea862397966ad9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a2951c5b5924d13bdea862397966ad9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36819 } } }
I20260926 07:34:17.946170 10926 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a2951c5b5924d13bdea862397966ad9 [sys.catalog]: This master's current role is: LEADER
I20260926 07:34:17.946592 10934 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:34:17.946547 10927 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a2951c5b5924d13bdea862397966ad9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a2951c5b5924d13bdea862397966ad9. Latest consensus state: current_term: 1 leader_uuid: "5a2951c5b5924d13bdea862397966ad9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a2951c5b5924d13bdea862397966ad9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36819 } } }
I20260926 07:34:17.946707 10927 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a2951c5b5924d13bdea862397966ad9 [sys.catalog]: This master's current role is: LEADER
I20260926 07:34:17.947213 22743 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:34:17.950570 10934 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:34:17.957567 10934 catalog_manager.cc:1347] Generated new cluster ID: 4b5b1ddcb2cf490fb34a072540f48391
I20260926 07:34:17.957647 10934 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:34:17.959311 10923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:17.949738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51627 (local address 127.0.0.1:36819)
0926 07:34:17.949857 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:17.949862 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:17.950170 (+ 308us) server_negotiation.cc:407] Connection header received
0926 07:34:17.950298 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:17.950301 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:17.950365 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:17.950457 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:17.951373 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:17.952207 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:17.953229 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:17.953498 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:17.953627 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:17.955203 (+ 1576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:17.955207 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:17.955212 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:17.955247 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:17.957132 (+ 1885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:17.957888 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:17.957892 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:17.957894 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:17.957995 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:17.958331 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:17.958335 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:17.958336 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:17.958588 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:17.958732 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:17.959009 (+ 277us) server_negotiation.cc:299] Negotiation successful
0926 07:34:17.959145 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 07:34:17.982216 10934 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:34:17.983299 10934 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:34:17.989454 10934 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5a2951c5b5924d13bdea862397966ad9: Generated new TSK 0
I20260926 07:34:17.990267 10934 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:34:17.995172 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 10852
2026-09-26T07:34:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1462 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407783529717-22743-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:34:18 dist-test-slave-wdjc krb5kdc[10951](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:34:18 dist-test-slave-wdjc krb5kdc[10951](info): set up 2 sockets
Sep 26 07:34:18 dist-test-slave-wdjc krb5kdc[10951](info): commencing operation
krb5kdc: starting...
W20260926 07:34:19.041018 22743 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407783529717-22743-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407783529717-22743-0/krb5kdc/test-user.keytab.
Sep 26 07:34:19 dist-test-slave-wdjc krb5kdc[10951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408059, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:34:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:34:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407783529717-22743-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407783529717-22743-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 07:34:19.171696 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407783529717-22743-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407783529717-22743-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407783529717-22743-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42237
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407783529717-22743-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42237
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:34:19.292191 10964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:34:19.292629 10964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:34:19.292749 10964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:34:19.297058 10964 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:34:19.297187 10964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:34:19.297226 10964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:34:19.297266 10964 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:34:19.297302 10964 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:34:19.303123 10964 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407783529717-22743-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407783529717-22743-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42237
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407783529717-22743-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42237
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10964
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407783529717-22743-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:34:19.304382 10964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:34:19.305495 10964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:34:19.314720 10972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:19.314690 10969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:34:19.314695 10964 server_base.cc:1034] running on GCE node
W20260926 07:34:19.314677 10970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:34:19.315618 10964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:34:19.316902 10964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:34:19.318090 10964 hybrid_clock.cc:648] HybridClock initialized: now 1790408059318076 us; error 31 us; skew 500 ppm
Sep 26 07:34:19 dist-test-slave-wdjc krb5kdc[10951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408059, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:34:19.325012 10964 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 07:34:19.326151 10964 webserver.cc:460] Webserver started at http://127.0.0.1:39037/ using document root <none> and password file <none>
I20260926 07:34:19.326804 10964 fs_manager.cc:359] Metadata directory not provided
I20260926 07:34:19.326908 10964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:34:19.327168 10964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:34:19.329576 10964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407783529717-22743-0/minicluster-data/master-0/data/instance:
uuid: "fa352e48b46c459896bbe10788f0a831"
format_stamp: "Formatted at 2026-09-26 07:34:19 on dist-test-slave-wdjc"
server_key: "f2b7ecb172b3922c553e6f2df01e3688"
server_key_iv: "dcbbb707945aaa61c9d7df0c376fca3c"
server_key_version: "encryptionkey@0"
I20260926 07:34:19.330224 10964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407783529717-22743-0/minicluster-data/master-0/wal/instance:
uuid: "fa352e48b46c459896bbe10788f0a831"
format_stamp: "Formatted at 2026-09-26 07:34:19 on dist-test-slave-wdjc"
server_key: "f2b7ecb172b3922c553e6f2df01e3688"
server_key_iv: "dcbbb707945aaa61c9d7df0c376fca3c"
server_key_version: "encryptionkey@0"
I20260926 07:34:19.334376 10964 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260926 07:34:19.337466 10979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:19.338953 10964 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 07:34:19.339133 10964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407783529717-22743-0/minicluster-data/master-0/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407783529717-22743-0/minicluster-data/master-0/wal
uuid: "fa352e48b46c459896bbe10788f0a831"
format_stamp: "Formatted at 2026-09-26 07:34:19 on dist-test-slave-wdjc"
server_key: "f2b7ecb172b3922c553e6f2df01e3688"
server_key_iv: "dcbbb707945aaa61c9d7df0c376fca3c"
server_key_version: "encryptionkey@0"
I20260926 07:34:19.339308 10964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407783529717-22743-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407783529717-22743-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407783529717-22743-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:34:19.366281 10964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:34:19.369972 10964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:34:19.370210 10964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:34:19.377797 10964 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42237
I20260926 07:34:19.377827 11031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42237 every 8 connection(s)
I20260926 07:34:19.378849 10964 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
I20260926 07:34:19.379364 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 10964
I20260926 07:34:19.379527 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407783529717-22743-0/minicluster-data/master-0/wal/instance
I20260926 07:34:19.379829 22743 external_mini_cluster.cc:1428] Setting key d89dc69b5899b8067f144507da341ca2
I20260926 07:34:19.382149 11032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:34:19.389288 11032 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa352e48b46c459896bbe10788f0a831: Bootstrap starting.
I20260926 07:34:19.389734 11035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:19.381383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55229 (local address 127.0.0.1:42237)
0926 07:34:19.381823 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:19.381841 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:19.381900 (+ 59us) server_negotiation.cc:407] Connection header received
0926 07:34:19.382650 (+ 750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:19.382669 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:19.383038 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:19.383363 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:19.383791 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:19.384935 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:19.386197 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:19.386692 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:19.386818 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:19.387791 (+ 973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:19.387796 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:19.387813 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:34:19.388221 (+ 408us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:19.388277 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:34:19.388319 (+ 42us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:19.388396 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:19.388632 (+ 236us) server_negotiation.cc:299] Negotiation successful
0926 07:34:19.388883 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":146,"threads_started":1}
I20260926 07:34:19.391975 11032 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fa352e48b46c459896bbe10788f0a831: Neither blocks nor log segments found. Creating new log.
I20260926 07:34:19.392920 22743 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:34:19.393028 11032 log.cc:824] T 00000000000000000000000000000000 P fa352e48b46c459896bbe10788f0a831: Log is configured to *not* fsync() on all Append() calls
I20260926 07:34:19.395363 11032 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa352e48b46c459896bbe10788f0a831: No bootstrap required, opened a new log
I20260926 07:34:19.399178 11035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:19.395171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45935 (local address 127.0.0.1:42237)
0926 07:34:19.395310 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:19.395317 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:19.395554 (+ 237us) server_negotiation.cc:407] Connection header received
0926 07:34:19.395724 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:19.395728 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:19.395778 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:19.395863 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:19.396137 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:19.396720 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:19.397549 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:19.397786 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:19.397898 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:19.398739 (+ 841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:19.398742 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:19.398745 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:34:19.398764 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:19.398791 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:34:19.398801 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:19.398879 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:19.398970 (+ 91us) server_negotiation.cc:299] Negotiation successful
0926 07:34:19.399028 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 07:34:19.399300 11032 raft_consensus.cc:348] T 00000000000000000000000000000000 P fa352e48b46c459896bbe10788f0a831 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa352e48b46c459896bbe10788f0a831" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42237 } }
I20260926 07:34:19.399669 11032 raft_consensus.cc:374] T 00000000000000000000000000000000 P fa352e48b46c459896bbe10788f0a831 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:34:19.399746 11032 raft_consensus.cc:729] T 00000000000000000000000000000000 P fa352e48b46c459896bbe10788f0a831 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa352e48b46c459896bbe10788f0a831, State: Initialized, Role: FOLLOWER
I20260926 07:34:19.400301 11032 consensus_queue.cc:260] T 00000000000000000000000000000000 P fa352e48b46c459896bbe10788f0a831 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa352e48b46c459896bbe10788f0a831" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42237 } }
I20260926 07:34:19.400456 11032 raft_consensus.cc:388] T 00000000000000000000000000000000 P fa352e48b46c459896bbe10788f0a831 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:34:19.400573 11032 raft_consensus.cc:482] T 00000000000000000000000000000000 P fa352e48b46c459896bbe10788f0a831 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:34:19.400682 11032 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fa352e48b46c459896bbe10788f0a831 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:34:19.401711 11032 raft_consensus.cc:504] T 00000000000000000000000000000000 P fa352e48b46c459896bbe10788f0a831 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa352e48b46c459896bbe10788f0a831" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42237 } }
I20260926 07:34:19.402128 11032 leader_election.cc:302] T 00000000000000000000000000000000 P fa352e48b46c459896bbe10788f0a831 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fa352e48b46c459896bbe10788f0a831; no voters:
I20260926 07:34:19.402473 11032 leader_election.cc:288] T 00000000000000000000000000000000 P fa352e48b46c459896bbe10788f0a831 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:34:19.402693 11042 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fa352e48b46c459896bbe10788f0a831 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:34:19.402958 11042 raft_consensus.cc:686] T 00000000000000000000000000000000 P fa352e48b46c459896bbe10788f0a831 [term 1 LEADER]: Becoming Leader. State: Replica: fa352e48b46c459896bbe10788f0a831, State: Running, Role: LEADER
I20260926 07:34:19.403306 11042 consensus_queue.cc:237] T 00000000000000000000000000000000 P fa352e48b46c459896bbe10788f0a831 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa352e48b46c459896bbe10788f0a831" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42237 } }
I20260926 07:34:19.403936 11032 sys_catalog.cc:564] T 00000000000000000000000000000000 P fa352e48b46c459896bbe10788f0a831 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:34:19.405103 11044 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa352e48b46c459896bbe10788f0a831 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fa352e48b46c459896bbe10788f0a831. Latest consensus state: current_term: 1 leader_uuid: "fa352e48b46c459896bbe10788f0a831" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa352e48b46c459896bbe10788f0a831" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42237 } } }
I20260926 07:34:19.405259 11044 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa352e48b46c459896bbe10788f0a831 [sys.catalog]: This master's current role is: LEADER
I20260926 07:34:19.405622 11043 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa352e48b46c459896bbe10788f0a831 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fa352e48b46c459896bbe10788f0a831" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa352e48b46c459896bbe10788f0a831" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42237 } } }
I20260926 07:34:19.405761 11043 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa352e48b46c459896bbe10788f0a831 [sys.catalog]: This master's current role is: LEADER
I20260926 07:34:19.406078 11051 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:34:19.409513 11051 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:34:19.416384 11051 catalog_manager.cc:1347] Generated new cluster ID: 74804b97ce9b4d878139644bfa145457
I20260926 07:34:19.416486 11051 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:34:19.427130 11051 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:34:19.428175 11051 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:34:19.442952 11051 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fa352e48b46c459896bbe10788f0a831: Generated new TSK 0
I20260926 07:34:19.443921 11051 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:34:19.456267 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 10964
2026-09-26T07:34:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1461 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407783529717-22743-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:34:19 dist-test-slave-wdjc krb5kdc[11063](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:34:19 dist-test-slave-wdjc krb5kdc[11063](info): set up 2 sockets
Sep 26 07:34:19 dist-test-slave-wdjc krb5kdc[11063](info): commencing operation
krb5kdc: starting...
W20260926 07:34:20.501001 22743 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407783529717-22743-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407783529717-22743-0/krb5kdc/test-user.keytab.
Sep 26 07:34:20 dist-test-slave-wdjc krb5kdc[11063](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408060, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:34:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:34:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407783529717-22743-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407783529717-22743-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 07:34:20.633217 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407783529717-22743-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407783529717-22743-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407783529717-22743-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36679
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407783529717-22743-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36679
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:34:20.752696 11076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:34:20.753036 11076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:34:20.753084 11076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:34:20.757521 11076 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:34:20.757618 11076 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:34:20.757642 11076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:34:20.757663 11076 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:34:20.757683 11076 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:34:20.763442 11076 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407783529717-22743-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407783529717-22743-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36679
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407783529717-22743-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36679
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11076
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407783529717-22743-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:34:20.764657 11076 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:34:20.765720 11076 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:34:20.775110 11082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:20.775152 11084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:20.775152 11081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:34:20.775445 11076 server_base.cc:1034] running on GCE node
I20260926 07:34:20.776078 11076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:34:20.777356 11076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:34:20.778553 11076 hybrid_clock.cc:648] HybridClock initialized: now 1790408060778514 us; error 53 us; skew 500 ppm
Sep 26 07:34:20 dist-test-slave-wdjc krb5kdc[11063](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408060, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:34:20.781641 11076 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 07:34:20.782687 11076 webserver.cc:460] Webserver started at http://127.0.0.1:39829/ using document root <none> and password file <none>
I20260926 07:34:20.783316 11076 fs_manager.cc:359] Metadata directory not provided
I20260926 07:34:20.783414 11076 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:34:20.783658 11076 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:34:20.785898 11076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407783529717-22743-0/minicluster-data/master-0/data/instance:
uuid: "08e4b7ef087e412380193cb33c58464f"
format_stamp: "Formatted at 2026-09-26 07:34:20 on dist-test-slave-wdjc"
server_key: "3c82d213252fc4ced4fc5ebf9fbb6fa9"
server_key_iv: "e06e6517ddcc87d02488263123d24450"
server_key_version: "encryptionkey@0"
I20260926 07:34:20.786500 11076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407783529717-22743-0/minicluster-data/master-0/wal/instance:
uuid: "08e4b7ef087e412380193cb33c58464f"
format_stamp: "Formatted at 2026-09-26 07:34:20 on dist-test-slave-wdjc"
server_key: "3c82d213252fc4ced4fc5ebf9fbb6fa9"
server_key_iv: "e06e6517ddcc87d02488263123d24450"
server_key_version: "encryptionkey@0"
I20260926 07:34:20.790735 11076 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 07:34:20.793618 11091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:20.795089 11076 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:34:20.795300 11076 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407783529717-22743-0/minicluster-data/master-0/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407783529717-22743-0/minicluster-data/master-0/wal
uuid: "08e4b7ef087e412380193cb33c58464f"
format_stamp: "Formatted at 2026-09-26 07:34:20 on dist-test-slave-wdjc"
server_key: "3c82d213252fc4ced4fc5ebf9fbb6fa9"
server_key_iv: "e06e6517ddcc87d02488263123d24450"
server_key_version: "encryptionkey@0"
I20260926 07:34:20.795449 11076 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407783529717-22743-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407783529717-22743-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407783529717-22743-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:34:20.800827 11076 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:34:20.804459 11076 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:34:20.804765 11076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:34:20.812930 11076 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36679
I20260926 07:34:20.812958 11143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36679 every 8 connection(s)
I20260926 07:34:20.814002 11076 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
I20260926 07:34:20.817142 11144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:34:20.820482 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 11076
I20260926 07:34:20.820701 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407783529717-22743-0/minicluster-data/master-0/wal/instance
I20260926 07:34:20.821025 22743 external_mini_cluster.cc:1428] Setting key 16a8f8390f05eee4fed67495b5914583
I20260926 07:34:20.823800 11144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08e4b7ef087e412380193cb33c58464f: Bootstrap starting.
I20260926 07:34:20.826587 11144 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 08e4b7ef087e412380193cb33c58464f: Neither blocks nor log segments found. Creating new log.
I20260926 07:34:20.826845 11147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:20.822718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38717 (local address 127.0.0.1:36679)
0926 07:34:20.823258 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:20.823273 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:20.823309 (+ 36us) server_negotiation.cc:407] Connection header received
0926 07:34:20.824028 (+ 719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:20.824046 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:20.824344 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:20.824732 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:20.825581 (+ 849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:20.825586 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:20.825599 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:34:20.825609 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:20.825664 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:34:20.825684 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:20.825840 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:20.825915 (+ 75us) server_negotiation.cc:299] Negotiation successful
0926 07:34:20.826079 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":158,"threads_started":1}
I20260926 07:34:20.827607 11144 log.cc:824] T 00000000000000000000000000000000 P 08e4b7ef087e412380193cb33c58464f: Log is configured to *not* fsync() on all Append() calls
I20260926 07:34:20.830070 11144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08e4b7ef087e412380193cb33c58464f: No bootstrap required, opened a new log
I20260926 07:34:20.830157 22743 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:34:20.834466 11147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:20.832421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51387 (local address 127.0.0.1:36679)
0926 07:34:20.832579 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:20.832586 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:20.832843 (+ 257us) server_negotiation.cc:407] Connection header received
0926 07:34:20.833008 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:20.833011 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:20.833055 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:20.833141 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:20.833994 (+ 853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:20.833998 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:20.834001 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:34:20.834013 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:20.834050 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:34:20.834061 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:20.834173 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:20.834278 (+ 105us) server_negotiation.cc:299] Negotiation successful
0926 07:34:20.834323 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 07:34:20.834774 11144 raft_consensus.cc:348] T 00000000000000000000000000000000 P 08e4b7ef087e412380193cb33c58464f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08e4b7ef087e412380193cb33c58464f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36679 } }
I20260926 07:34:20.835144 11144 raft_consensus.cc:374] T 00000000000000000000000000000000 P 08e4b7ef087e412380193cb33c58464f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:34:20.835243 11144 raft_consensus.cc:729] T 00000000000000000000000000000000 P 08e4b7ef087e412380193cb33c58464f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08e4b7ef087e412380193cb33c58464f, State: Initialized, Role: FOLLOWER
I20260926 07:34:20.835781 11144 consensus_queue.cc:260] T 00000000000000000000000000000000 P 08e4b7ef087e412380193cb33c58464f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08e4b7ef087e412380193cb33c58464f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36679 } }
I20260926 07:34:20.835958 11144 raft_consensus.cc:388] T 00000000000000000000000000000000 P 08e4b7ef087e412380193cb33c58464f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:34:20.836035 11144 raft_consensus.cc:482] T 00000000000000000000000000000000 P 08e4b7ef087e412380193cb33c58464f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:34:20.836154 11144 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 08e4b7ef087e412380193cb33c58464f [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:34:20.837353 11144 raft_consensus.cc:504] T 00000000000000000000000000000000 P 08e4b7ef087e412380193cb33c58464f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08e4b7ef087e412380193cb33c58464f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36679 } }
I20260926 07:34:20.837764 11144 leader_election.cc:302] T 00000000000000000000000000000000 P 08e4b7ef087e412380193cb33c58464f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 08e4b7ef087e412380193cb33c58464f; no voters:
I20260926 07:34:20.838172 11144 leader_election.cc:288] T 00000000000000000000000000000000 P 08e4b7ef087e412380193cb33c58464f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:34:20.838357 11154 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 08e4b7ef087e412380193cb33c58464f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:34:20.838721 11154 raft_consensus.cc:686] T 00000000000000000000000000000000 P 08e4b7ef087e412380193cb33c58464f [term 1 LEADER]: Becoming Leader. State: Replica: 08e4b7ef087e412380193cb33c58464f, State: Running, Role: LEADER
I20260926 07:34:20.839118 11154 consensus_queue.cc:237] T 00000000000000000000000000000000 P 08e4b7ef087e412380193cb33c58464f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08e4b7ef087e412380193cb33c58464f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36679 } }
I20260926 07:34:20.839787 11144 sys_catalog.cc:564] T 00000000000000000000000000000000 P 08e4b7ef087e412380193cb33c58464f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:34:20.841085 11156 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08e4b7ef087e412380193cb33c58464f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 08e4b7ef087e412380193cb33c58464f. Latest consensus state: current_term: 1 leader_uuid: "08e4b7ef087e412380193cb33c58464f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08e4b7ef087e412380193cb33c58464f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36679 } } }
I20260926 07:34:20.841142 11155 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08e4b7ef087e412380193cb33c58464f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "08e4b7ef087e412380193cb33c58464f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08e4b7ef087e412380193cb33c58464f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36679 } } }
I20260926 07:34:20.841316 11155 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08e4b7ef087e412380193cb33c58464f [sys.catalog]: This master's current role is: LEADER
I20260926 07:34:20.841251 11156 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08e4b7ef087e412380193cb33c58464f [sys.catalog]: This master's current role is: LEADER
W20260926 07:34:20.845980 11169 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 08e4b7ef087e412380193cb33c58464f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:34:20.846094 11169 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:34:20.846161 11163 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:34:20.846946 11163 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:34:20.852363 11163 catalog_manager.cc:1347] Generated new cluster ID: 6643489ac09b4fec8f08c7365e6ab4d7
I20260926 07:34:20.852463 11163 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:34:20.863628 11163 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:34:20.864050 11163 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:34:20.873389 11163 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 08e4b7ef087e412380193cb33c58464f: Generated new TSK 0
I20260926 07:34:20.874480 11163 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:34:20.885728 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 11076
2026-09-26T07:34:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1429 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407783529717-22743-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:34:20 dist-test-slave-wdjc krb5kdc[11175](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:34:20 dist-test-slave-wdjc krb5kdc[11175](info): set up 2 sockets
Sep 26 07:34:20 dist-test-slave-wdjc krb5kdc[11175](info): commencing operation
krb5kdc: starting...
W20260926 07:34:21.965314 22743 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407783529717-22743-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407783529717-22743-0/krb5kdc/test-user.keytab.
Sep 26 07:34:22 dist-test-slave-wdjc krb5kdc[11175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408062, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:34:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:34:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407783529717-22743-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407783529717-22743-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 07:34:22.096312 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407783529717-22743-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407783529717-22743-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407783529717-22743-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45149
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407783529717-22743-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45149
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:34:22.214516 11188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:34:22.214942 11188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:34:22.215039 11188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:34:22.219476 11188 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:34:22.219640 11188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:34:22.219687 11188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:34:22.219731 11188 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:34:22.219776 11188 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:34:22.226944 11188 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407783529717-22743-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407783529717-22743-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45149
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407783529717-22743-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11188
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407783529717-22743-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:34:22.228672 11188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:34:22.230145 11188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:34:22.239771 11194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:22.239774 11196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:22.239804 11193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:34:22.240203 11188 server_base.cc:1034] running on GCE node
I20260926 07:34:22.241093 11188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:34:22.242308 11188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:34:22.243479 11188 hybrid_clock.cc:648] HybridClock initialized: now 1790408062243460 us; error 41 us; skew 500 ppm
Sep 26 07:34:22 dist-test-slave-wdjc krb5kdc[11175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408062, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:34:22.246630 11188 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 07:34:22.247676 11188 webserver.cc:460] Webserver started at http://127.0.0.1:35179/ using document root <none> and password file <none>
I20260926 07:34:22.248319 11188 fs_manager.cc:359] Metadata directory not provided
I20260926 07:34:22.248421 11188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:34:22.248752 11188 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:34:22.251003 11188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407783529717-22743-0/minicluster-data/master-0/data/instance:
uuid: "61f2b9a87b614531917e1eacfc0610af"
format_stamp: "Formatted at 2026-09-26 07:34:22 on dist-test-slave-wdjc"
server_key: "baf9a27fdd2e7e10dba8f3217c3df681"
server_key_iv: "0c83a5d7e7b5e5ef67f60c4b0a05ea7b"
server_key_version: "encryptionkey@0"
I20260926 07:34:22.251629 11188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407783529717-22743-0/minicluster-data/master-0/wal/instance:
uuid: "61f2b9a87b614531917e1eacfc0610af"
format_stamp: "Formatted at 2026-09-26 07:34:22 on dist-test-slave-wdjc"
server_key: "baf9a27fdd2e7e10dba8f3217c3df681"
server_key_iv: "0c83a5d7e7b5e5ef67f60c4b0a05ea7b"
server_key_version: "encryptionkey@0"
I20260926 07:34:22.255885 11188 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260926 07:34:22.258759 11203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:22.260192 11188 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 07:34:22.260373 11188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407783529717-22743-0/minicluster-data/master-0/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407783529717-22743-0/minicluster-data/master-0/wal
uuid: "61f2b9a87b614531917e1eacfc0610af"
format_stamp: "Formatted at 2026-09-26 07:34:22 on dist-test-slave-wdjc"
server_key: "baf9a27fdd2e7e10dba8f3217c3df681"
server_key_iv: "0c83a5d7e7b5e5ef67f60c4b0a05ea7b"
server_key_version: "encryptionkey@0"
I20260926 07:34:22.260515 11188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407783529717-22743-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407783529717-22743-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407783529717-22743-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:34:22.281972 11188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:34:22.285439 11188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:34:22.285674 11188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:34:22.293272 11188 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45149
I20260926 07:34:22.293308 11255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45149 every 8 connection(s)
I20260926 07:34:22.294394 11188 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
I20260926 07:34:22.297482 11256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:34:22.303536 11256 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61f2b9a87b614531917e1eacfc0610af: Bootstrap starting.
I20260926 07:34:22.304040 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 11188
I20260926 07:34:22.304196 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407783529717-22743-0/minicluster-data/master-0/wal/instance
I20260926 07:34:22.304515 22743 external_mini_cluster.cc:1428] Setting key 90d38855f704543af182d90b5617dcab
I20260926 07:34:22.306567 11256 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 61f2b9a87b614531917e1eacfc0610af: Neither blocks nor log segments found. Creating new log.
I20260926 07:34:22.307519 11256 log.cc:824] T 00000000000000000000000000000000 P 61f2b9a87b614531917e1eacfc0610af: Log is configured to *not* fsync() on all Append() calls
I20260926 07:34:22.309964 11256 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61f2b9a87b614531917e1eacfc0610af: No bootstrap required, opened a new log
I20260926 07:34:22.314544 11256 raft_consensus.cc:348] T 00000000000000000000000000000000 P 61f2b9a87b614531917e1eacfc0610af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61f2b9a87b614531917e1eacfc0610af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45149 } }
I20260926 07:34:22.314885 11256 raft_consensus.cc:374] T 00000000000000000000000000000000 P 61f2b9a87b614531917e1eacfc0610af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:34:22.314935 11256 raft_consensus.cc:729] T 00000000000000000000000000000000 P 61f2b9a87b614531917e1eacfc0610af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61f2b9a87b614531917e1eacfc0610af, State: Initialized, Role: FOLLOWER
Sep 26 07:34:22 dist-test-slave-wdjc krb5kdc[11175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408062, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 07:34:22.315441 11256 consensus_queue.cc:260] T 00000000000000000000000000000000 P 61f2b9a87b614531917e1eacfc0610af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61f2b9a87b614531917e1eacfc0610af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45149 } }
I20260926 07:34:22.315618 11256 raft_consensus.cc:388] T 00000000000000000000000000000000 P 61f2b9a87b614531917e1eacfc0610af [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:34:22.315702 11256 raft_consensus.cc:482] T 00000000000000000000000000000000 P 61f2b9a87b614531917e1eacfc0610af [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:34:22.315807 11256 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 61f2b9a87b614531917e1eacfc0610af [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:34:22.317229 11256 raft_consensus.cc:504] T 00000000000000000000000000000000 P 61f2b9a87b614531917e1eacfc0610af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61f2b9a87b614531917e1eacfc0610af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45149 } }
I20260926 07:34:22.317654 11256 leader_election.cc:302] T 00000000000000000000000000000000 P 61f2b9a87b614531917e1eacfc0610af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 61f2b9a87b614531917e1eacfc0610af; no voters:
I20260926 07:34:22.318082 11256 leader_election.cc:288] T 00000000000000000000000000000000 P 61f2b9a87b614531917e1eacfc0610af [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:34:22.318212 11261 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 61f2b9a87b614531917e1eacfc0610af [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:34:22.318462 11261 raft_consensus.cc:686] T 00000000000000000000000000000000 P 61f2b9a87b614531917e1eacfc0610af [term 1 LEADER]: Becoming Leader. State: Replica: 61f2b9a87b614531917e1eacfc0610af, State: Running, Role: LEADER
I20260926 07:34:22.318851 11261 consensus_queue.cc:237] T 00000000000000000000000000000000 P 61f2b9a87b614531917e1eacfc0610af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61f2b9a87b614531917e1eacfc0610af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45149 } }
I20260926 07:34:22.319581 11256 sys_catalog.cc:564] T 00000000000000000000000000000000 P 61f2b9a87b614531917e1eacfc0610af [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:34:22.320276 11261 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61f2b9a87b614531917e1eacfc0610af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 61f2b9a87b614531917e1eacfc0610af. Latest consensus state: current_term: 1 leader_uuid: "61f2b9a87b614531917e1eacfc0610af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61f2b9a87b614531917e1eacfc0610af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45149 } } }
I20260926 07:34:22.320438 11261 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61f2b9a87b614531917e1eacfc0610af [sys.catalog]: This master's current role is: LEADER
I20260926 07:34:22.320996 11269 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:34:22.321103 11262 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61f2b9a87b614531917e1eacfc0610af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "61f2b9a87b614531917e1eacfc0610af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61f2b9a87b614531917e1eacfc0610af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45149 } } }
I20260926 07:34:22.321280 11262 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61f2b9a87b614531917e1eacfc0610af [sys.catalog]: This master's current role is: LEADER
I20260926 07:34:22.321897 11259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:22.306050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.29:52179 (local address 127.0.0.1:45149)
0926 07:34:22.306520 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:22.306541 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:22.306587 (+ 46us) server_negotiation.cc:407] Connection header received
0926 07:34:22.307555 (+ 968us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:22.307581 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:22.308043 (+ 462us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:22.308395 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:22.309215 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:22.310701 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:22.311637 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:22.312382 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:22.312580 (+ 198us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:22.315679 (+ 3099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:22.315687 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:22.315705 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:22.315749 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:22.317890 (+ 2141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:22.318731 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:22.318737 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:22.318743 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:22.318887 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:22.319307 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:22.319311 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:22.319313 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:22.319883 (+ 570us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:22.320150 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:22.320483 (+ 333us) server_negotiation.cc:299] Negotiation successful
0926 07:34:22.320799 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":136,"threads_started":1}
I20260926 07:34:22.325104 11269 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:34:22.325632 22743 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:34:22.331928 11269 catalog_manager.cc:1347] Generated new cluster ID: d29bba09cc06455c8ce40e0452d56c0e
I20260926 07:34:22.332029 11269 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:34:22.338843 11269 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:34:22.338964 11259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:22.328007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.29:35387 (local address 127.0.0.1:45149)
0926 07:34:22.328126 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:22.328141 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:22.328451 (+ 310us) server_negotiation.cc:407] Connection header received
0926 07:34:22.328637 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:22.328641 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:22.328697 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:22.328793 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:22.329546 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:22.330345 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:22.331619 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:22.332010 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:22.332094 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:22.334497 (+ 2403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:22.334501 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:22.334505 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:22.334538 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:22.336471 (+ 1933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:22.337186 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:22.337190 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:22.337192 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:22.337298 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:22.337814 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:22.337817 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:22.337819 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:22.338073 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:22.338267 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:22.338590 (+ 323us) server_negotiation.cc:299] Negotiation successful
0926 07:34:22.338706 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 07:34:22.340229 11269 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:34:22.353058 11269 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 61f2b9a87b614531917e1eacfc0610af: Generated new TSK 0
I20260926 07:34:22.354074 11269 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:34:22.359664 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 11188
W20260926 07:34:22.365756 11173 connection.cc:535] client connection to 127.0.0.1:45149 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45149): Connection reset by peer (error 104)
2026-09-26T07:34:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1473 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407783529717-22743-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:34:22 dist-test-slave-wdjc krb5kdc[11287](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:34:22 dist-test-slave-wdjc krb5kdc[11287](info): set up 2 sockets
Sep 26 07:34:22 dist-test-slave-wdjc krb5kdc[11287](info): commencing operation
krb5kdc: starting...
W20260926 07:34:23.417178 22743 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407783529717-22743-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407783529717-22743-0/krb5kdc/test-user.keytab.
Sep 26 07:34:23 dist-test-slave-wdjc krb5kdc[11287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408063, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:34:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:34:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407783529717-22743-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407783529717-22743-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 07:34:23.548642 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407783529717-22743-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407783529717-22743-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407783529717-22743-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34739
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407783529717-22743-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34739
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:34:23.664361 11300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:34:23.664785 11300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:34:23.664881 11300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:34:23.669340 11300 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:34:23.669480 11300 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:34:23.669519 11300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:34:23.669559 11300 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:34:23.669595 11300 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:34:23.675451 11300 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407783529717-22743-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407783529717-22743-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34739
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407783529717-22743-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34739
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11300
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407783529717-22743-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:34:23.676769 11300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:34:23.677953 11300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:34:23.687279 11306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:23.687296 11308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:23.687299 11305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:34:23.687397 11300 server_base.cc:1034] running on GCE node
I20260926 07:34:23.688225 11300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:34:23.689522 11300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:34:23.690733 11300 hybrid_clock.cc:648] HybridClock initialized: now 1790408063690713 us; error 38 us; skew 500 ppm
Sep 26 07:34:23 dist-test-slave-wdjc krb5kdc[11287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408063, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:34:23.693738 11300 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 07:34:23.694762 11300 webserver.cc:460] Webserver started at http://127.0.0.1:36005/ using document root <none> and password file <none>
I20260926 07:34:23.695372 11300 fs_manager.cc:359] Metadata directory not provided
I20260926 07:34:23.695488 11300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:34:23.695736 11300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:34:23.698004 11300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407783529717-22743-0/minicluster-data/master-0/data/instance:
uuid: "d0b7fb77b1f54315b3c0d8fb949047c0"
format_stamp: "Formatted at 2026-09-26 07:34:23 on dist-test-slave-wdjc"
server_key: "888770607279bd1fdffedee12095d4ed"
server_key_iv: "359864bba05e8e6ddb6a923d962b9177"
server_key_version: "encryptionkey@0"
I20260926 07:34:23.698619 11300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407783529717-22743-0/minicluster-data/master-0/wal/instance:
uuid: "d0b7fb77b1f54315b3c0d8fb949047c0"
format_stamp: "Formatted at 2026-09-26 07:34:23 on dist-test-slave-wdjc"
server_key: "888770607279bd1fdffedee12095d4ed"
server_key_iv: "359864bba05e8e6ddb6a923d962b9177"
server_key_version: "encryptionkey@0"
I20260926 07:34:23.702692 11300 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 07:34:23.705433 11315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:23.706857 11300 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:34:23.707043 11300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407783529717-22743-0/minicluster-data/master-0/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407783529717-22743-0/minicluster-data/master-0/wal
uuid: "d0b7fb77b1f54315b3c0d8fb949047c0"
format_stamp: "Formatted at 2026-09-26 07:34:23 on dist-test-slave-wdjc"
server_key: "888770607279bd1fdffedee12095d4ed"
server_key_iv: "359864bba05e8e6ddb6a923d962b9177"
server_key_version: "encryptionkey@0"
I20260926 07:34:23.707226 11300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407783529717-22743-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407783529717-22743-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407783529717-22743-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:34:23.725572 11300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:34:23.729040 11300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:34:23.729307 11300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:34:23.737030 11300 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34739
I20260926 07:34:23.737097 11367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34739 every 8 connection(s)
I20260926 07:34:23.738119 11300 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
I20260926 07:34:23.741269 11368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:34:23.746804 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 11300
I20260926 07:34:23.746965 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407783529717-22743-0/minicluster-data/master-0/wal/instance
I20260926 07:34:23.747241 22743 external_mini_cluster.cc:1428] Setting key a2ad5a4a58539735f5d4f4cb0abffec7
I20260926 07:34:23.747929 11368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0b7fb77b1f54315b3c0d8fb949047c0: Bootstrap starting.
I20260926 07:34:23.750528 11368 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d0b7fb77b1f54315b3c0d8fb949047c0: Neither blocks nor log segments found. Creating new log.
I20260926 07:34:23.751422 11368 log.cc:824] T 00000000000000000000000000000000 P d0b7fb77b1f54315b3c0d8fb949047c0: Log is configured to *not* fsync() on all Append() calls
I20260926 07:34:23.753594 11368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0b7fb77b1f54315b3c0d8fb949047c0: No bootstrap required, opened a new log
I20260926 07:34:23.757141 11371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:23.748784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.29:60369 (local address 127.0.0.1:34739)
0926 07:34:23.749236 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:23.749252 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:23.749289 (+ 37us) server_negotiation.cc:407] Connection header received
0926 07:34:23.749970 (+ 681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:23.749988 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:23.750361 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:23.750718 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:23.751106 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:23.752176 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:23.753443 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:23.754019 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:23.754213 (+ 194us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:23.755179 (+ 966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:23.755184 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:23.755203 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:34:23.755616 (+ 413us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:23.755674 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:34:23.755708 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:23.755857 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:23.756050 (+ 193us) server_negotiation.cc:299] Negotiation successful
0926 07:34:23.756202 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":146,"threads_started":1}
I20260926 07:34:23.758045 11368 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0b7fb77b1f54315b3c0d8fb949047c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0b7fb77b1f54315b3c0d8fb949047c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34739 } }
I20260926 07:34:23.758309 11368 raft_consensus.cc:374] T 00000000000000000000000000000000 P d0b7fb77b1f54315b3c0d8fb949047c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:34:23.758387 11368 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0b7fb77b1f54315b3c0d8fb949047c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0b7fb77b1f54315b3c0d8fb949047c0, State: Initialized, Role: FOLLOWER
I20260926 07:34:23.758828 11368 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0b7fb77b1f54315b3c0d8fb949047c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0b7fb77b1f54315b3c0d8fb949047c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34739 } }
I20260926 07:34:23.758981 11368 raft_consensus.cc:388] T 00000000000000000000000000000000 P d0b7fb77b1f54315b3c0d8fb949047c0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:34:23.759059 11368 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0b7fb77b1f54315b3c0d8fb949047c0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:34:23.759194 11368 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0b7fb77b1f54315b3c0d8fb949047c0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:34:23.760406 22743 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:34:23.760449 11368 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0b7fb77b1f54315b3c0d8fb949047c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0b7fb77b1f54315b3c0d8fb949047c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34739 } }
I20260926 07:34:23.760835 11368 leader_election.cc:302] T 00000000000000000000000000000000 P d0b7fb77b1f54315b3c0d8fb949047c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d0b7fb77b1f54315b3c0d8fb949047c0; no voters:
I20260926 07:34:23.761214 11368 leader_election.cc:288] T 00000000000000000000000000000000 P d0b7fb77b1f54315b3c0d8fb949047c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:34:23.761281 11373 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d0b7fb77b1f54315b3c0d8fb949047c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:34:23.761482 11373 raft_consensus.cc:686] T 00000000000000000000000000000000 P d0b7fb77b1f54315b3c0d8fb949047c0 [term 1 LEADER]: Becoming Leader. State: Replica: d0b7fb77b1f54315b3c0d8fb949047c0, State: Running, Role: LEADER
I20260926 07:34:23.761868 11373 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0b7fb77b1f54315b3c0d8fb949047c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0b7fb77b1f54315b3c0d8fb949047c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34739 } }
I20260926 07:34:23.762633 11368 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0b7fb77b1f54315b3c0d8fb949047c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:34:23.763830 11374 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0b7fb77b1f54315b3c0d8fb949047c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d0b7fb77b1f54315b3c0d8fb949047c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0b7fb77b1f54315b3c0d8fb949047c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34739 } } }
I20260926 07:34:23.763968 11374 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0b7fb77b1f54315b3c0d8fb949047c0 [sys.catalog]: This master's current role is: LEADER
I20260926 07:34:23.764185 11374 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0b7fb77b1f54315b3c0d8fb949047c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0b7fb77b1f54315b3c0d8fb949047c0. Latest consensus state: current_term: 1 leader_uuid: "d0b7fb77b1f54315b3c0d8fb949047c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0b7fb77b1f54315b3c0d8fb949047c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34739 } } }
I20260926 07:34:23.764263 11374 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0b7fb77b1f54315b3c0d8fb949047c0 [sys.catalog]: This master's current role is: LEADER
I20260926 07:34:23.764722 11387 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:34:23.768229 11371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:23.763000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.29:44211 (local address 127.0.0.1:34739)
0926 07:34:23.763133 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:23.763136 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:23.763410 (+ 274us) server_negotiation.cc:407] Connection header received
0926 07:34:23.763620 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:23.763623 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:23.763677 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:23.763735 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:23.764199 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:23.764988 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:23.766225 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:23.766588 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:23.766745 (+ 157us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:23.767700 (+ 955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:23.767704 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:23.767707 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:34:23.767722 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:23.767757 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:34:23.767769 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:23.767872 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:23.768026 (+ 154us) server_negotiation.cc:299] Negotiation successful
0926 07:34:23.768064 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 07:34:23.768378 11387 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:34:23.775414 11387 catalog_manager.cc:1347] Generated new cluster ID: 9b6e653c7c5344948def02cadebdb930
I20260926 07:34:23.775533 11387 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:34:23.799081 11387 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:34:23.800151 11387 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:34:23.809974 11387 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d0b7fb77b1f54315b3c0d8fb949047c0: Generated new TSK 0
I20260926 07:34:23.810842 11387 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:34:23.819525 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 11300
W20260926 07:34:23.824867 11285 connection.cc:535] client connection to 127.0.0.1:34739 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34739): Connection reset by peer (error 104)
2026-09-26T07:34:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1459 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407783529717-22743-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:34:23 dist-test-slave-wdjc krb5kdc[11399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:34:23 dist-test-slave-wdjc krb5kdc[11399](info): set up 2 sockets
Sep 26 07:34:23 dist-test-slave-wdjc krb5kdc[11399](info): commencing operation
krb5kdc: starting...
W20260926 07:34:24.884995 22743 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407783529717-22743-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407783529717-22743-0/krb5kdc/test-user.keytab.
Sep 26 07:34:24 dist-test-slave-wdjc krb5kdc[11399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408064, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:34:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:34:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407783529717-22743-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407783529717-22743-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 07:34:25.015972 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407783529717-22743-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407783529717-22743-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407783529717-22743-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43733
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407783529717-22743-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43733
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:34:25.128609 11412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:34:25.128912 11412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:34:25.128957 11412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:34:25.133170 11412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:34:25.133244 11412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:34:25.133265 11412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:34:25.133281 11412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:34:25.133297 11412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:34:25.138943 11412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407783529717-22743-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407783529717-22743-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43733
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407783529717-22743-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407783529717-22743-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:34:25.140152 11412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:34:25.141270 11412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:34:25.150338 11420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:34:25.150619 11412 server_base.cc:1034] running on GCE node
W20260926 07:34:25.150338 11418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:25.150338 11417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:34:25.151391 11412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:34:25.152596 11412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:34:25.153788 11412 hybrid_clock.cc:648] HybridClock initialized: now 1790408065153768 us; error 39 us; skew 500 ppm
Sep 26 07:34:25 dist-test-slave-wdjc krb5kdc[11399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408065, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:34:25.160804 11412 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 07:34:25.161912 11412 webserver.cc:460] Webserver started at http://127.0.0.1:33353/ using document root <none> and password file <none>
I20260926 07:34:25.162537 11412 fs_manager.cc:359] Metadata directory not provided
I20260926 07:34:25.162629 11412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:34:25.162876 11412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:34:25.165109 11412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407783529717-22743-0/minicluster-data/master-0/data/instance:
uuid: "b675dc49082e42bf94676218204f1c25"
format_stamp: "Formatted at 2026-09-26 07:34:25 on dist-test-slave-wdjc"
server_key: "2eb80aae0c0979a45ebbc49c1bc40538"
server_key_iv: "0951a3ece0ebf3ecb55ea3d0deae8c21"
server_key_version: "encryptionkey@0"
I20260926 07:34:25.165699 11412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407783529717-22743-0/minicluster-data/master-0/wal/instance:
uuid: "b675dc49082e42bf94676218204f1c25"
format_stamp: "Formatted at 2026-09-26 07:34:25 on dist-test-slave-wdjc"
server_key: "2eb80aae0c0979a45ebbc49c1bc40538"
server_key_iv: "0951a3ece0ebf3ecb55ea3d0deae8c21"
server_key_version: "encryptionkey@0"
I20260926 07:34:25.169817 11412 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 07:34:25.172602 11427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:25.174073 11412 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:34:25.174256 11412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407783529717-22743-0/minicluster-data/master-0/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407783529717-22743-0/minicluster-data/master-0/wal
uuid: "b675dc49082e42bf94676218204f1c25"
format_stamp: "Formatted at 2026-09-26 07:34:25 on dist-test-slave-wdjc"
server_key: "2eb80aae0c0979a45ebbc49c1bc40538"
server_key_iv: "0951a3ece0ebf3ecb55ea3d0deae8c21"
server_key_version: "encryptionkey@0"
I20260926 07:34:25.174417 11412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407783529717-22743-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407783529717-22743-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407783529717-22743-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:34:25.179668 11412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:34:25.183331 11412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:34:25.183578 11412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:34:25.191473 11412 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43733
I20260926 07:34:25.191502 11479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43733 every 8 connection(s)
I20260926 07:34:25.192514 11412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
I20260926 07:34:25.193056 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 11412
I20260926 07:34:25.193228 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407783529717-22743-0/minicluster-data/master-0/wal/instance
I20260926 07:34:25.193548 22743 external_mini_cluster.cc:1428] Setting key 049220842623538e7491eeb631ee2f12
I20260926 07:34:25.195896 11480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:34:25.199170 11483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:25.195098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.29:48727 (local address 127.0.0.1:43733)
0926 07:34:25.195545 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:25.195564 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:25.195608 (+ 44us) server_negotiation.cc:407] Connection header received
0926 07:34:25.196336 (+ 728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:25.196354 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:25.196714 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:25.197003 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:25.197891 (+ 888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:25.197895 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:25.197907 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:34:25.197917 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:25.197974 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:34:25.197993 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:25.198106 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:25.198239 (+ 133us) server_negotiation.cc:299] Negotiation successful
0926 07:34:25.198427 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":148,"threads_started":1}
I20260926 07:34:25.202502 22743 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:34:25.203043 11480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b675dc49082e42bf94676218204f1c25: Bootstrap starting.
I20260926 07:34:25.205783 11480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b675dc49082e42bf94676218204f1c25: Neither blocks nor log segments found. Creating new log.
I20260926 07:34:25.206641 11480 log.cc:824] T 00000000000000000000000000000000 P b675dc49082e42bf94676218204f1c25: Log is configured to *not* fsync() on all Append() calls
I20260926 07:34:25.206712 11483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:25.204815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.29:35523 (local address 127.0.0.1:43733)
0926 07:34:25.204961 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:25.204965 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:25.205161 (+ 196us) server_negotiation.cc:407] Connection header received
0926 07:34:25.205272 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:25.205274 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:25.205314 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:25.205394 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:25.206242 (+ 848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:25.206251 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:25.206253 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:34:25.206263 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:25.206291 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:34:25.206302 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:25.206402 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:25.206501 (+ 99us) server_negotiation.cc:299] Negotiation successful
0926 07:34:25.206547 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 07:34:25.209156 11480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b675dc49082e42bf94676218204f1c25: No bootstrap required, opened a new log
I20260926 07:34:25.213060 11480 raft_consensus.cc:348] T 00000000000000000000000000000000 P b675dc49082e42bf94676218204f1c25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b675dc49082e42bf94676218204f1c25" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43733 } }
I20260926 07:34:25.213357 11480 raft_consensus.cc:374] T 00000000000000000000000000000000 P b675dc49082e42bf94676218204f1c25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:34:25.213440 11480 raft_consensus.cc:729] T 00000000000000000000000000000000 P b675dc49082e42bf94676218204f1c25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b675dc49082e42bf94676218204f1c25, State: Initialized, Role: FOLLOWER
I20260926 07:34:25.213933 11480 consensus_queue.cc:260] T 00000000000000000000000000000000 P b675dc49082e42bf94676218204f1c25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b675dc49082e42bf94676218204f1c25" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43733 } }
I20260926 07:34:25.214087 11480 raft_consensus.cc:388] T 00000000000000000000000000000000 P b675dc49082e42bf94676218204f1c25 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:34:25.214157 11480 raft_consensus.cc:482] T 00000000000000000000000000000000 P b675dc49082e42bf94676218204f1c25 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:34:25.214290 11480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b675dc49082e42bf94676218204f1c25 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:34:25.215324 11480 raft_consensus.cc:504] T 00000000000000000000000000000000 P b675dc49082e42bf94676218204f1c25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b675dc49082e42bf94676218204f1c25" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43733 } }
I20260926 07:34:25.215711 11480 leader_election.cc:302] T 00000000000000000000000000000000 P b675dc49082e42bf94676218204f1c25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b675dc49082e42bf94676218204f1c25; no voters:
I20260926 07:34:25.216091 11480 leader_election.cc:288] T 00000000000000000000000000000000 P b675dc49082e42bf94676218204f1c25 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:34:25.216259 11490 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b675dc49082e42bf94676218204f1c25 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:34:25.216521 11490 raft_consensus.cc:686] T 00000000000000000000000000000000 P b675dc49082e42bf94676218204f1c25 [term 1 LEADER]: Becoming Leader. State: Replica: b675dc49082e42bf94676218204f1c25, State: Running, Role: LEADER
I20260926 07:34:25.216888 11490 consensus_queue.cc:237] T 00000000000000000000000000000000 P b675dc49082e42bf94676218204f1c25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b675dc49082e42bf94676218204f1c25" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43733 } }
I20260926 07:34:25.217456 11480 sys_catalog.cc:564] T 00000000000000000000000000000000 P b675dc49082e42bf94676218204f1c25 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:34:25.218612 11492 sys_catalog.cc:455] T 00000000000000000000000000000000 P b675dc49082e42bf94676218204f1c25 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b675dc49082e42bf94676218204f1c25. Latest consensus state: current_term: 1 leader_uuid: "b675dc49082e42bf94676218204f1c25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b675dc49082e42bf94676218204f1c25" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43733 } } }
I20260926 07:34:25.218755 11492 sys_catalog.cc:458] T 00000000000000000000000000000000 P b675dc49082e42bf94676218204f1c25 [sys.catalog]: This master's current role is: LEADER
I20260926 07:34:25.219161 11491 sys_catalog.cc:455] T 00000000000000000000000000000000 P b675dc49082e42bf94676218204f1c25 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b675dc49082e42bf94676218204f1c25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b675dc49082e42bf94676218204f1c25" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43733 } } }
I20260926 07:34:25.219259 11491 sys_catalog.cc:458] T 00000000000000000000000000000000 P b675dc49082e42bf94676218204f1c25 [sys.catalog]: This master's current role is: LEADER
W20260926 07:34:25.222891 11505 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b675dc49082e42bf94676218204f1c25: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:34:25.222988 11505 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:34:25.223078 11499 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:34:25.223848 11499 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:34:25.230227 11499 catalog_manager.cc:1347] Generated new cluster ID: d9678f254a454204a1f477d2671634e1
I20260926 07:34:25.230324 11499 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:34:25.248920 11499 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:34:25.249354 11499 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:34:25.263060 11499 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b675dc49082e42bf94676218204f1c25: Generated new TSK 0
I20260926 07:34:25.264037 11499 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:34:25.278460 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 11412
2026-09-26T07:34:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1459 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407783529717-22743-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:34:25 dist-test-slave-wdjc krb5kdc[11511](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:34:25 dist-test-slave-wdjc krb5kdc[11511](info): set up 2 sockets
Sep 26 07:34:25 dist-test-slave-wdjc krb5kdc[11511](info): commencing operation
krb5kdc: starting...
W20260926 07:34:26.325192 22743 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407783529717-22743-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407783529717-22743-0/krb5kdc/test-user.keytab.
Sep 26 07:34:26 dist-test-slave-wdjc krb5kdc[11511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408066, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:34:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:34:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407783529717-22743-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407783529717-22743-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 07:34:26.456590 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407783529717-22743-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407783529717-22743-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407783529717-22743-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37633
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407783529717-22743-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37633
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:34:26.576980 11524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:34:26.577359 11524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:34:26.577456 11524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:34:26.581743 11524 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:34:26.581890 11524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:34:26.581930 11524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:34:26.581969 11524 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:34:26.582000 11524 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:34:26.587764 11524 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407783529717-22743-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407783529717-22743-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37633
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407783529717-22743-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37633
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11524
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407783529717-22743-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:34:26.589071 11524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:34:26.590186 11524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:34:26.599318 11532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:26.599318 11530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:26.599368 11529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:34:26.599624 11524 server_base.cc:1034] running on GCE node
I20260926 07:34:26.600420 11524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:34:26.601708 11524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:34:26.602918 11524 hybrid_clock.cc:648] HybridClock initialized: now 1790408066602906 us; error 31 us; skew 500 ppm
Sep 26 07:34:26 dist-test-slave-wdjc krb5kdc[11511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408066, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:34:26.606025 11524 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 07:34:26.607054 11524 webserver.cc:460] Webserver started at http://127.0.0.1:35409/ using document root <none> and password file <none>
I20260926 07:34:26.607679 11524 fs_manager.cc:359] Metadata directory not provided
I20260926 07:34:26.607782 11524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:34:26.608040 11524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:34:26.610484 11524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407783529717-22743-0/minicluster-data/master-0/data/instance:
uuid: "c053f9d5cbb647928671e0ec05073e07"
format_stamp: "Formatted at 2026-09-26 07:34:26 on dist-test-slave-wdjc"
server_key: "3970e339a9ef208c0497a5eaaee0664f"
server_key_iv: "d830399ef945bdbcf1d25874b3d19d64"
server_key_version: "encryptionkey@0"
I20260926 07:34:26.611128 11524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407783529717-22743-0/minicluster-data/master-0/wal/instance:
uuid: "c053f9d5cbb647928671e0ec05073e07"
format_stamp: "Formatted at 2026-09-26 07:34:26 on dist-test-slave-wdjc"
server_key: "3970e339a9ef208c0497a5eaaee0664f"
server_key_iv: "d830399ef945bdbcf1d25874b3d19d64"
server_key_version: "encryptionkey@0"
I20260926 07:34:26.616693 11524 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260926 07:34:26.619486 11539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:26.621006 11524 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:34:26.621168 11524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407783529717-22743-0/minicluster-data/master-0/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407783529717-22743-0/minicluster-data/master-0/wal
uuid: "c053f9d5cbb647928671e0ec05073e07"
format_stamp: "Formatted at 2026-09-26 07:34:26 on dist-test-slave-wdjc"
server_key: "3970e339a9ef208c0497a5eaaee0664f"
server_key_iv: "d830399ef945bdbcf1d25874b3d19d64"
server_key_version: "encryptionkey@0"
I20260926 07:34:26.621304 11524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407783529717-22743-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407783529717-22743-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407783529717-22743-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:34:26.640172 11524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:34:26.643787 11524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:34:26.644028 11524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:34:26.651582 11524 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37633
I20260926 07:34:26.651603 11591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37633 every 8 connection(s)
I20260926 07:34:26.652681 11524 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
I20260926 07:34:26.654457 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 11524
I20260926 07:34:26.654613 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407783529717-22743-0/minicluster-data/master-0/wal/instance
I20260926 07:34:26.654938 22743 external_mini_cluster.cc:1428] Setting key 135ac91383c50aa62ebd8fc084ca4c65
I20260926 07:34:26.656077 11592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:34:26.662351 11592 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c053f9d5cbb647928671e0ec05073e07: Bootstrap starting.
Sep 26 07:34:26 dist-test-slave-wdjc krb5kdc[11511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408066, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 07:34:26.665197 11592 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c053f9d5cbb647928671e0ec05073e07: Neither blocks nor log segments found. Creating new log.
I20260926 07:34:26.666095 11592 log.cc:824] T 00000000000000000000000000000000 P c053f9d5cbb647928671e0ec05073e07: Log is configured to *not* fsync() on all Append() calls
I20260926 07:34:26.668213 11592 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c053f9d5cbb647928671e0ec05073e07: No bootstrap required, opened a new log
I20260926 07:34:26.670379 11595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:26.656516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.29:58193 (local address 127.0.0.1:37633)
0926 07:34:26.657039 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:26.657055 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:26.657093 (+ 38us) server_negotiation.cc:407] Connection header received
0926 07:34:26.657767 (+ 674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:26.657785 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:26.658166 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:26.658562 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:26.659460 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:26.660601 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:26.661582 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:26.662140 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:26.662327 (+ 187us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:26.665333 (+ 3006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:26.665340 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:26.665360 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:26.665404 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:26.667261 (+ 1857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:26.667853 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:26.667858 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:26.667863 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:26.667977 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:26.668330 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:26.668336 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:26.668342 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:26.668822 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:26.669060 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:26.669377 (+ 317us) server_negotiation.cc:299] Negotiation successful
0926 07:34:26.669617 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":164,"threads_started":1}
I20260926 07:34:26.672111 11592 raft_consensus.cc:348] T 00000000000000000000000000000000 P c053f9d5cbb647928671e0ec05073e07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c053f9d5cbb647928671e0ec05073e07" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37633 } }
I20260926 07:34:26.672459 11592 raft_consensus.cc:374] T 00000000000000000000000000000000 P c053f9d5cbb647928671e0ec05073e07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:34:26.672569 11592 raft_consensus.cc:729] T 00000000000000000000000000000000 P c053f9d5cbb647928671e0ec05073e07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c053f9d5cbb647928671e0ec05073e07, State: Initialized, Role: FOLLOWER
I20260926 07:34:26.673130 11592 consensus_queue.cc:260] T 00000000000000000000000000000000 P c053f9d5cbb647928671e0ec05073e07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c053f9d5cbb647928671e0ec05073e07" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37633 } }
I20260926 07:34:26.673305 11592 raft_consensus.cc:388] T 00000000000000000000000000000000 P c053f9d5cbb647928671e0ec05073e07 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:34:26.673395 11592 raft_consensus.cc:482] T 00000000000000000000000000000000 P c053f9d5cbb647928671e0ec05073e07 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:34:26.673497 11592 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c053f9d5cbb647928671e0ec05073e07 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:34:26.673696 22743 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:34:26.674655 11592 raft_consensus.cc:504] T 00000000000000000000000000000000 P c053f9d5cbb647928671e0ec05073e07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c053f9d5cbb647928671e0ec05073e07" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37633 } }
I20260926 07:34:26.675063 11592 leader_election.cc:302] T 00000000000000000000000000000000 P c053f9d5cbb647928671e0ec05073e07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c053f9d5cbb647928671e0ec05073e07; no voters:
I20260926 07:34:26.675465 11592 leader_election.cc:288] T 00000000000000000000000000000000 P c053f9d5cbb647928671e0ec05073e07 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:34:26.675532 11601 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c053f9d5cbb647928671e0ec05073e07 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:34:26.675745 11601 raft_consensus.cc:686] T 00000000000000000000000000000000 P c053f9d5cbb647928671e0ec05073e07 [term 1 LEADER]: Becoming Leader. State: Replica: c053f9d5cbb647928671e0ec05073e07, State: Running, Role: LEADER
I20260926 07:34:26.676108 11601 consensus_queue.cc:237] T 00000000000000000000000000000000 P c053f9d5cbb647928671e0ec05073e07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c053f9d5cbb647928671e0ec05073e07" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37633 } }
I20260926 07:34:26.676895 11592 sys_catalog.cc:564] T 00000000000000000000000000000000 P c053f9d5cbb647928671e0ec05073e07 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:34:26.677362 11601 sys_catalog.cc:455] T 00000000000000000000000000000000 P c053f9d5cbb647928671e0ec05073e07 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c053f9d5cbb647928671e0ec05073e07. Latest consensus state: current_term: 1 leader_uuid: "c053f9d5cbb647928671e0ec05073e07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c053f9d5cbb647928671e0ec05073e07" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37633 } } }
I20260926 07:34:26.677480 11601 sys_catalog.cc:458] T 00000000000000000000000000000000 P c053f9d5cbb647928671e0ec05073e07 [sys.catalog]: This master's current role is: LEADER
I20260926 07:34:26.677685 11601 sys_catalog.cc:455] T 00000000000000000000000000000000 P c053f9d5cbb647928671e0ec05073e07 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c053f9d5cbb647928671e0ec05073e07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c053f9d5cbb647928671e0ec05073e07" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37633 } } }
I20260926 07:34:26.677852 11606 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:34:26.677856 11601 sys_catalog.cc:458] T 00000000000000000000000000000000 P c053f9d5cbb647928671e0ec05073e07 [sys.catalog]: This master's current role is: LEADER
I20260926 07:34:26.681430 11606 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:34:26.688655 11595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:26.675867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.29:36243 (local address 127.0.0.1:37633)
0926 07:34:26.675994 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:26.675999 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:26.678491 (+ 2492us) server_negotiation.cc:407] Connection header received
0926 07:34:26.679291 (+ 800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:26.679294 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:26.679350 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:26.679443 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:26.680300 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:26.681071 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:26.682271 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:26.682628 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:26.682740 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:26.684630 (+ 1890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:26.684634 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:26.684637 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:26.684669 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:26.686513 (+ 1844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:26.687137 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:26.687140 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:26.687141 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:26.687222 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:26.687594 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:26.687598 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:26.687601 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:26.687844 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:26.688012 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:26.688306 (+ 294us) server_negotiation.cc:299] Negotiation successful
0926 07:34:26.688422 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 07:34:26.694269 11606 catalog_manager.cc:1347] Generated new cluster ID: 86da81f3a0cc434e87fb525fc98396de
I20260926 07:34:26.694437 11606 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:34:26.702337 11606 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:34:26.703341 11606 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:34:26.711994 11606 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c053f9d5cbb647928671e0ec05073e07: Generated new TSK 0
I20260926 07:34:26.712877 11606 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:34:26.718350 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 11524
W20260926 07:34:26.723851 11509 connection.cc:535] client connection to 127.0.0.1:37633 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37633): Connection reset by peer (error 104)
2026-09-26T07:34:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1439 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407783529717-22743-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:34:26 dist-test-slave-wdjc krb5kdc[11623](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:34:26 dist-test-slave-wdjc krb5kdc[11623](info): set up 2 sockets
Sep 26 07:34:26 dist-test-slave-wdjc krb5kdc[11623](info): commencing operation
krb5kdc: starting...
W20260926 07:34:27.789047 22743 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407783529717-22743-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407783529717-22743-0/krb5kdc/test-user.keytab.
Sep 26 07:34:27 dist-test-slave-wdjc krb5kdc[11623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408067, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:34:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:34:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407783529717-22743-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407783529717-22743-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 07:34:27.921417 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407783529717-22743-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407783529717-22743-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407783529717-22743-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46371
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407783529717-22743-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46371
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:34:28.042106 11636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:34:28.042449 11636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:34:28.042496 11636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:34:28.046819 11636 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:34:28.046914 11636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:34:28.046936 11636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:34:28.046953 11636 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:34:28.046970 11636 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:34:28.052793 11636 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407783529717-22743-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407783529717-22743-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46371
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407783529717-22743-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46371
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11636
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407783529717-22743-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:34:28.053992 11636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:34:28.055051 11636 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:34:28.064383 11641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:28.064371 11642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:28.064371 11644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:34:28.064932 11636 server_base.cc:1034] running on GCE node
I20260926 07:34:28.065536 11636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:34:28.066824 11636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:34:28.068046 11636 hybrid_clock.cc:648] HybridClock initialized: now 1790408068068031 us; error 58 us; skew 500 ppm
Sep 26 07:34:28 dist-test-slave-wdjc krb5kdc[11623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408068, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:34:28.071297 11636 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 07:34:28.072348 11636 webserver.cc:460] Webserver started at http://127.0.0.1:37751/ using document root <none> and password file <none>
I20260926 07:34:28.073051 11636 fs_manager.cc:359] Metadata directory not provided
I20260926 07:34:28.073143 11636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:34:28.073381 11636 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:34:28.075673 11636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407783529717-22743-0/minicluster-data/master-0/data/instance:
uuid: "84f548535d27405fb650a139115e101d"
format_stamp: "Formatted at 2026-09-26 07:34:28 on dist-test-slave-wdjc"
server_key: "144aed19a830aa90f1c29c6d6c20fa9d"
server_key_iv: "7d433b63170503926c733a5b6f626f91"
server_key_version: "encryptionkey@0"
I20260926 07:34:28.076329 11636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407783529717-22743-0/minicluster-data/master-0/wal/instance:
uuid: "84f548535d27405fb650a139115e101d"
format_stamp: "Formatted at 2026-09-26 07:34:28 on dist-test-slave-wdjc"
server_key: "144aed19a830aa90f1c29c6d6c20fa9d"
server_key_iv: "7d433b63170503926c733a5b6f626f91"
server_key_version: "encryptionkey@0"
I20260926 07:34:28.080881 11636 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 07:34:28.084401 11651 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:28.086644 11636 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 07:34:28.086885 11636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407783529717-22743-0/minicluster-data/master-0/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407783529717-22743-0/minicluster-data/master-0/wal
uuid: "84f548535d27405fb650a139115e101d"
format_stamp: "Formatted at 2026-09-26 07:34:28 on dist-test-slave-wdjc"
server_key: "144aed19a830aa90f1c29c6d6c20fa9d"
server_key_iv: "7d433b63170503926c733a5b6f626f91"
server_key_version: "encryptionkey@0"
I20260926 07:34:28.087055 11636 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407783529717-22743-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407783529717-22743-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407783529717-22743-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:34:28.107473 11636 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:34:28.111222 11636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:34:28.111476 11636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:34:28.119769 11636 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46371
I20260926 07:34:28.119796 11703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46371 every 8 connection(s)
I20260926 07:34:28.120922 11636 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
I20260926 07:34:28.124042 11704 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:34:28.129487 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 11636
I20260926 07:34:28.129621 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407783529717-22743-0/minicluster-data/master-0/wal/instance
I20260926 07:34:28.129973 22743 external_mini_cluster.cc:1428] Setting key 3e60c733821a80badbe8b647460ad0b7
I20260926 07:34:28.131135 11704 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84f548535d27405fb650a139115e101d: Bootstrap starting.
I20260926 07:34:28.134209 11704 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 84f548535d27405fb650a139115e101d: Neither blocks nor log segments found. Creating new log.
I20260926 07:34:28.135200 11704 log.cc:824] T 00000000000000000000000000000000 P 84f548535d27405fb650a139115e101d: Log is configured to *not* fsync() on all Append() calls
I20260926 07:34:28.137742 11704 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84f548535d27405fb650a139115e101d: No bootstrap required, opened a new log
I20260926 07:34:28.142416 11707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:28.131721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.29:43291 (local address 127.0.0.1:46371)
0926 07:34:28.132192 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:28.132216 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:28.132261 (+ 45us) server_negotiation.cc:407] Connection header received
0926 07:34:28.133262 (+ 1001us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:28.133290 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:28.133777 (+ 487us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:28.134195 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:28.134694 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:28.136148 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:28.137655 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:28.138423 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:28.138606 (+ 183us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:28.139854 (+ 1248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:28.139861 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:28.139881 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:34:28.140472 (+ 591us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:28.140583 (+ 111us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:34:28.140618 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:28.140768 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:28.141011 (+ 243us) server_negotiation.cc:299] Negotiation successful
0926 07:34:28.141177 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":159,"threads_started":1}
I20260926 07:34:28.143191 11704 raft_consensus.cc:348] T 00000000000000000000000000000000 P 84f548535d27405fb650a139115e101d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84f548535d27405fb650a139115e101d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46371 } }
I20260926 07:34:28.143500 11704 raft_consensus.cc:374] T 00000000000000000000000000000000 P 84f548535d27405fb650a139115e101d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:34:28.143558 11704 raft_consensus.cc:729] T 00000000000000000000000000000000 P 84f548535d27405fb650a139115e101d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84f548535d27405fb650a139115e101d, State: Initialized, Role: FOLLOWER
I20260926 07:34:28.144125 11704 consensus_queue.cc:260] T 00000000000000000000000000000000 P 84f548535d27405fb650a139115e101d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84f548535d27405fb650a139115e101d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46371 } }
I20260926 07:34:28.144268 11704 raft_consensus.cc:388] T 00000000000000000000000000000000 P 84f548535d27405fb650a139115e101d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:34:28.144362 11704 raft_consensus.cc:482] T 00000000000000000000000000000000 P 84f548535d27405fb650a139115e101d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:34:28.144491 11704 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 84f548535d27405fb650a139115e101d [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:34:28.145774 11704 raft_consensus.cc:504] T 00000000000000000000000000000000 P 84f548535d27405fb650a139115e101d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84f548535d27405fb650a139115e101d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46371 } }
I20260926 07:34:28.146020 22743 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:34:28.146209 11704 leader_election.cc:302] T 00000000000000000000000000000000 P 84f548535d27405fb650a139115e101d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 84f548535d27405fb650a139115e101d; no voters:
I20260926 07:34:28.146582 11704 leader_election.cc:288] T 00000000000000000000000000000000 P 84f548535d27405fb650a139115e101d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:34:28.146742 11709 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 84f548535d27405fb650a139115e101d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:34:28.146991 11709 raft_consensus.cc:686] T 00000000000000000000000000000000 P 84f548535d27405fb650a139115e101d [term 1 LEADER]: Becoming Leader. State: Replica: 84f548535d27405fb650a139115e101d, State: Running, Role: LEADER
I20260926 07:34:28.147483 11709 consensus_queue.cc:237] T 00000000000000000000000000000000 P 84f548535d27405fb650a139115e101d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84f548535d27405fb650a139115e101d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46371 } }
I20260926 07:34:28.148010 11704 sys_catalog.cc:564] T 00000000000000000000000000000000 P 84f548535d27405fb650a139115e101d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:34:28.149479 11710 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84f548535d27405fb650a139115e101d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "84f548535d27405fb650a139115e101d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84f548535d27405fb650a139115e101d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46371 } } }
I20260926 07:34:28.149622 11710 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84f548535d27405fb650a139115e101d [sys.catalog]: This master's current role is: LEADER
I20260926 07:34:28.149997 11711 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84f548535d27405fb650a139115e101d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 84f548535d27405fb650a139115e101d. Latest consensus state: current_term: 1 leader_uuid: "84f548535d27405fb650a139115e101d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84f548535d27405fb650a139115e101d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46371 } } }
I20260926 07:34:28.150096 11711 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84f548535d27405fb650a139115e101d [sys.catalog]: This master's current role is: LEADER
W20260926 07:34:28.153553 11729 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 84f548535d27405fb650a139115e101d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:34:28.153699 11729 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:34:28.153803 11724 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:34:28.154585 11724 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:34:28.157689 11707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:28.148546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.29:37689 (local address 127.0.0.1:46371)
0926 07:34:28.148677 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:28.148681 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:28.152970 (+ 4289us) server_negotiation.cc:407] Connection header received
0926 07:34:28.153158 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:28.153162 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:28.153224 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:28.153329 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:28.153660 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:28.154470 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:28.155639 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:28.155992 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:28.156107 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:28.157148 (+ 1041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:28.157152 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:28.157155 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:34:28.157172 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:28.157209 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:34:28.157223 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:28.157324 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:28.157490 (+ 166us) server_negotiation.cc:299] Negotiation successful
0926 07:34:28.157533 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 07:34:28.161471 11724 catalog_manager.cc:1347] Generated new cluster ID: 23e1f7251acc4bd68dda8296ef357af1
I20260926 07:34:28.161583 11724 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:34:28.179443 11724 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:34:28.180622 11724 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:34:28.188374 11724 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 84f548535d27405fb650a139115e101d: Generated new TSK 0
I20260926 07:34:28.189288 11724 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:34:28.194844 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 11636
W20260926 07:34:28.201362 11621 connection.cc:535] client connection to 127.0.0.1:46371 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46371): Connection reset by peer (error 104)
2026-09-26T07:34:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1477 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407783529717-22743-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:34:28 dist-test-slave-wdjc krb5kdc[11735](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:34:28 dist-test-slave-wdjc krb5kdc[11735](info): set up 2 sockets
Sep 26 07:34:28 dist-test-slave-wdjc krb5kdc[11735](info): commencing operation
krb5kdc: starting...
W20260926 07:34:29.249817 22743 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407783529717-22743-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407783529717-22743-0/krb5kdc/test-user.keytab.
Sep 26 07:34:29 dist-test-slave-wdjc krb5kdc[11735](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408069, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:34:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:34:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407783529717-22743-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407783529717-22743-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 07:34:29.379299 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407783529717-22743-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407783529717-22743-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407783529717-22743-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38053
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407783529717-22743-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38053
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:34:29.494136 11748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:34:29.494450 11748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:34:29.494495 11748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:34:29.498961 11748 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:34:29.499047 11748 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:34:29.499068 11748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:34:29.499085 11748 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:34:29.499101 11748 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:34:29.505061 11748 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407783529717-22743-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407783529717-22743-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38053
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407783529717-22743-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38053
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11748
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407783529717-22743-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:34:29.506361 11748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:34:29.507498 11748 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:34:29.517050 11753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:29.517050 11754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:34:29.517158 11748 server_base.cc:1034] running on GCE node
W20260926 07:34:29.517047 11756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:34:29.518085 11748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:34:29.519414 11748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:34:29.520658 11748 hybrid_clock.cc:648] HybridClock initialized: now 1790408069520624 us; error 57 us; skew 500 ppm
Sep 26 07:34:29 dist-test-slave-wdjc krb5kdc[11735](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408069, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:34:29.523926 11748 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 07:34:29.525250 11748 webserver.cc:460] Webserver started at http://127.0.0.1:38661/ using document root <none> and password file <none>
I20260926 07:34:29.526176 11748 fs_manager.cc:359] Metadata directory not provided
I20260926 07:34:29.526290 11748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:34:29.526563 11748 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:34:29.529600 11748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407783529717-22743-0/minicluster-data/master-0/data/instance:
uuid: "b3c04adddbdf417589fefb60839d6f15"
format_stamp: "Formatted at 2026-09-26 07:34:29 on dist-test-slave-wdjc"
server_key: "17aebe46ef39eda777be5800a925429c"
server_key_iv: "73a451c93a6fc13f9f9b250901bfcb25"
server_key_version: "encryptionkey@0"
I20260926 07:34:29.530272 11748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407783529717-22743-0/minicluster-data/master-0/wal/instance:
uuid: "b3c04adddbdf417589fefb60839d6f15"
format_stamp: "Formatted at 2026-09-26 07:34:29 on dist-test-slave-wdjc"
server_key: "17aebe46ef39eda777be5800a925429c"
server_key_iv: "73a451c93a6fc13f9f9b250901bfcb25"
server_key_version: "encryptionkey@0"
I20260926 07:34:29.534474 11748 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260926 07:34:29.537318 11763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:29.538708 11748 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:34:29.538885 11748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407783529717-22743-0/minicluster-data/master-0/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407783529717-22743-0/minicluster-data/master-0/wal
uuid: "b3c04adddbdf417589fefb60839d6f15"
format_stamp: "Formatted at 2026-09-26 07:34:29 on dist-test-slave-wdjc"
server_key: "17aebe46ef39eda777be5800a925429c"
server_key_iv: "73a451c93a6fc13f9f9b250901bfcb25"
server_key_version: "encryptionkey@0"
I20260926 07:34:29.539016 11748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407783529717-22743-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407783529717-22743-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407783529717-22743-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:34:29.544121 11748 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:34:29.547731 11748 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:34:29.547981 11748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:34:29.556347 11748 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38053
I20260926 07:34:29.556375 11815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38053 every 8 connection(s)
I20260926 07:34:29.557502 11748 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
I20260926 07:34:29.560721 11816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:34:29.566727 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 11748
I20260926 07:34:29.566913 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407783529717-22743-0/minicluster-data/master-0/wal/instance
I20260926 07:34:29.567003 11816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3c04adddbdf417589fefb60839d6f15: Bootstrap starting.
I20260926 07:34:29.567225 22743 external_mini_cluster.cc:1428] Setting key 3d84946cc513c78d5d94722a830f68b6
I20260926 07:34:29.570065 11816 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b3c04adddbdf417589fefb60839d6f15: Neither blocks nor log segments found. Creating new log.
I20260926 07:34:29.570923 11816 log.cc:824] T 00000000000000000000000000000000 P b3c04adddbdf417589fefb60839d6f15: Log is configured to *not* fsync() on all Append() calls
I20260926 07:34:29.573238 11816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3c04adddbdf417589fefb60839d6f15: No bootstrap required, opened a new log
I20260926 07:34:29.573477 11819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:29.568907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.29:46173 (local address 127.0.0.1:38053)
0926 07:34:29.569432 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:29.569447 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:29.569486 (+ 39us) server_negotiation.cc:407] Connection header received
0926 07:34:29.570283 (+ 797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:29.570304 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:29.570621 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:29.571001 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:29.571991 (+ 990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:29.571997 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:29.572023 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:34:29.572037 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:29.572101 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:34:29.572135 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:29.572279 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:29.572404 (+ 125us) server_negotiation.cc:299] Negotiation successful
0926 07:34:29.572613 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":176,"threads_started":1}
I20260926 07:34:29.577050 22743 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:34:29.577973 11816 raft_consensus.cc:348] T 00000000000000000000000000000000 P b3c04adddbdf417589fefb60839d6f15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3c04adddbdf417589fefb60839d6f15" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38053 } }
I20260926 07:34:29.578261 11816 raft_consensus.cc:374] T 00000000000000000000000000000000 P b3c04adddbdf417589fefb60839d6f15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:34:29.578320 11816 raft_consensus.cc:729] T 00000000000000000000000000000000 P b3c04adddbdf417589fefb60839d6f15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3c04adddbdf417589fefb60839d6f15, State: Initialized, Role: FOLLOWER
I20260926 07:34:29.578972 11816 consensus_queue.cc:260] T 00000000000000000000000000000000 P b3c04adddbdf417589fefb60839d6f15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3c04adddbdf417589fefb60839d6f15" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38053 } }
I20260926 07:34:29.579176 11816 raft_consensus.cc:388] T 00000000000000000000000000000000 P b3c04adddbdf417589fefb60839d6f15 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:34:29.579259 11816 raft_consensus.cc:482] T 00000000000000000000000000000000 P b3c04adddbdf417589fefb60839d6f15 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:34:29.579386 11816 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b3c04adddbdf417589fefb60839d6f15 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:34:29.580777 11816 raft_consensus.cc:504] T 00000000000000000000000000000000 P b3c04adddbdf417589fefb60839d6f15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3c04adddbdf417589fefb60839d6f15" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38053 } }
I20260926 07:34:29.581211 11816 leader_election.cc:302] T 00000000000000000000000000000000 P b3c04adddbdf417589fefb60839d6f15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b3c04adddbdf417589fefb60839d6f15; no voters:
I20260926 07:34:29.581420 11819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:29.579298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.29:50065 (local address 127.0.0.1:38053)
0926 07:34:29.579440 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:29.579447 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:29.579650 (+ 203us) server_negotiation.cc:407] Connection header received
0926 07:34:29.579760 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:29.579766 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:29.579811 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:29.579898 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:29.580815 (+ 917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:29.580819 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:29.580821 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:34:29.580834 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:29.580866 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:34:29.580881 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:29.580973 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:29.581130 (+ 157us) server_negotiation.cc:299] Negotiation successful
0926 07:34:29.581173 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 07:34:29.581678 11816 leader_election.cc:288] T 00000000000000000000000000000000 P b3c04adddbdf417589fefb60839d6f15 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:34:29.581774 11826 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b3c04adddbdf417589fefb60839d6f15 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:34:29.582010 11826 raft_consensus.cc:686] T 00000000000000000000000000000000 P b3c04adddbdf417589fefb60839d6f15 [term 1 LEADER]: Becoming Leader. State: Replica: b3c04adddbdf417589fefb60839d6f15, State: Running, Role: LEADER
I20260926 07:34:29.582345 11826 consensus_queue.cc:237] T 00000000000000000000000000000000 P b3c04adddbdf417589fefb60839d6f15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3c04adddbdf417589fefb60839d6f15" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38053 } }
I20260926 07:34:29.583153 11816 sys_catalog.cc:564] T 00000000000000000000000000000000 P b3c04adddbdf417589fefb60839d6f15 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:34:29.583559 11826 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3c04adddbdf417589fefb60839d6f15 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3c04adddbdf417589fefb60839d6f15. Latest consensus state: current_term: 1 leader_uuid: "b3c04adddbdf417589fefb60839d6f15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3c04adddbdf417589fefb60839d6f15" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38053 } } }
I20260926 07:34:29.583683 11826 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3c04adddbdf417589fefb60839d6f15 [sys.catalog]: This master's current role is: LEADER
I20260926 07:34:29.584041 11827 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3c04adddbdf417589fefb60839d6f15 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b3c04adddbdf417589fefb60839d6f15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3c04adddbdf417589fefb60839d6f15" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38053 } } }
I20260926 07:34:29.584051 11830 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:34:29.584184 11827 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3c04adddbdf417589fefb60839d6f15 [sys.catalog]: This master's current role is: LEADER
I20260926 07:34:29.587617 11830 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:34:29.593704 11830 catalog_manager.cc:1347] Generated new cluster ID: 340a53d5aea1422c9009252905d1db17
I20260926 07:34:29.593792 11830 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:34:29.605636 11830 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:34:29.606043 11830 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:34:29.618009 11830 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b3c04adddbdf417589fefb60839d6f15: Generated new TSK 0
I20260926 07:34:29.619010 11830 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:34:29.626479 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 11748
2026-09-26T07:34:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1430 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407783529717-22743-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:34:29 dist-test-slave-wdjc krb5kdc[11847](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:34:29 dist-test-slave-wdjc krb5kdc[11847](info): set up 2 sockets
Sep 26 07:34:29 dist-test-slave-wdjc krb5kdc[11847](info): commencing operation
krb5kdc: starting...
W20260926 07:34:30.673300 22743 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407783529717-22743-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407783529717-22743-0/krb5kdc/test-user.keytab.
Sep 26 07:34:30 dist-test-slave-wdjc krb5kdc[11847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408070, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:34:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:34:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.53.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.53.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.53.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab.
Entry for principal kudu/127.22.53.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab.
I20260926 07:34:30.802255 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407783529717-22743-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407783529717-22743-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407783529717-22743-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.53.254:42313
--webserver_interface=127.22.53.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab
--principal=kudu/127.22.53.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:38473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.53.254:42313
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:34:30.919636 11860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:34:30.920001 11860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:34:30.920087 11860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:34:30.924387 11860 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:34:30.924510 11860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:34:30.924546 11860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:34:30.924612 11860 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:34:30.924682 11860 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:34:30.930255 11860 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:38473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407783529717-22743-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407783529717-22743-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.53.254:42313
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab
--principal=kudu/127.22.53.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.53.254:42313
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.53.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11860
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407783529717-22743-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:34:30.931484 11860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:34:30.932574 11860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:34:30.941256 11865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:30.941258 11866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:30.941258 11868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:34:30.941604 11860 server_base.cc:1034] running on GCE node
I20260926 07:34:30.942286 11860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:34:30.943480 11860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:34:30.944710 11860 hybrid_clock.cc:648] HybridClock initialized: now 1790408070944680 us; error 47 us; skew 500 ppm
Sep 26 07:34:30 dist-test-slave-wdjc krb5kdc[11847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408070, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:34:30.951738 11860 init.cc:377] Logged in from keytab as kudu/127.22.53.254@KRBTEST.COM (short username kudu)
I20260926 07:34:30.952971 11860 webserver.cc:460] Webserver started at http://127.22.53.254:36605/ using document root <none> and password file <none>
I20260926 07:34:30.953631 11860 fs_manager.cc:359] Metadata directory not provided
I20260926 07:34:30.953743 11860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:34:30.954002 11860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:34:30.956353 11860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407783529717-22743-0/minicluster-data/master-0/data/instance:
uuid: "7d071aff74644741aaa9e6d5140628c8"
format_stamp: "Formatted at 2026-09-26 07:34:30 on dist-test-slave-wdjc"
server_key: "aed2daaf8b79f9431696a40e905064de"
server_key_iv: "d83163fd96133381d53aaad171724350"
server_key_version: "encryptionkey@0"
I20260926 07:34:30.957088 11860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407783529717-22743-0/minicluster-data/master-0/wal/instance:
uuid: "7d071aff74644741aaa9e6d5140628c8"
format_stamp: "Formatted at 2026-09-26 07:34:30 on dist-test-slave-wdjc"
server_key: "aed2daaf8b79f9431696a40e905064de"
server_key_iv: "d83163fd96133381d53aaad171724350"
server_key_version: "encryptionkey@0"
I20260926 07:34:30.961526 11860 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 07:34:30.966702 11875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:30.969100 11860 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20260926 07:34:30.969381 11860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407783529717-22743-0/minicluster-data/master-0/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407783529717-22743-0/minicluster-data/master-0/wal
uuid: "7d071aff74644741aaa9e6d5140628c8"
format_stamp: "Formatted at 2026-09-26 07:34:30 on dist-test-slave-wdjc"
server_key: "aed2daaf8b79f9431696a40e905064de"
server_key_iv: "d83163fd96133381d53aaad171724350"
server_key_version: "encryptionkey@0"
I20260926 07:34:30.969570 11860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407783529717-22743-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407783529717-22743-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407783529717-22743-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:34:31.006388 11860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:34:31.010324 11860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:34:31.010576 11860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:34:31.018785 11860 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.254:42313
I20260926 07:34:31.018828 11927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.254:42313 every 8 connection(s)
I20260926 07:34:31.019884 11860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
I20260926 07:34:31.020340 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 11860
I20260926 07:34:31.020516 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407783529717-22743-0/minicluster-data/master-0/wal/instance
I20260926 07:34:31.020848 22743 external_mini_cluster.cc:1428] Setting key 84f8f085a153d3693cbc8e24ba7a4ef4
I20260926 07:34:31.023523 11928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:34:31.030937 11928 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d071aff74644741aaa9e6d5140628c8: Bootstrap starting.
Sep 26 07:34:31 dist-test-slave-wdjc krb5kdc[11847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408070, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.53.254@KRBTEST.COM
I20260926 07:34:31.034096 11928 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7d071aff74644741aaa9e6d5140628c8: Neither blocks nor log segments found. Creating new log.
I20260926 07:34:31.035063 11928 log.cc:824] T 00000000000000000000000000000000 P 7d071aff74644741aaa9e6d5140628c8: Log is configured to *not* fsync() on all Append() calls
I20260926 07:34:31.037778 11928 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d071aff74644741aaa9e6d5140628c8: No bootstrap required, opened a new log
I20260926 07:34:31.038558 11931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:31.022425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47931 (local address 127.22.53.254:42313)
0926 07:34:31.022890 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:31.022909 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:31.022953 (+ 44us) server_negotiation.cc:407] Connection header received
0926 07:34:31.023838 (+ 885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:31.023863 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:31.024336 (+ 473us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:31.024758 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:31.025610 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:31.027133 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:31.028080 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:31.028861 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:31.029034 (+ 173us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:31.032831 (+ 3797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:31.032838 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:31.032855 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:31.032898 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:31.034975 (+ 2077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:31.035666 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:31.035671 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:31.035676 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:31.035796 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:31.036194 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:31.036197 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:31.036199 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:31.036775 (+ 576us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:31.036994 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:31.037310 (+ 316us) server_negotiation.cc:299] Negotiation successful
0926 07:34:31.037550 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":126,"threads_started":1}
I20260926 07:34:31.041890 22743 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:34:31.042394 11928 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7d071aff74644741aaa9e6d5140628c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d071aff74644741aaa9e6d5140628c8" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 42313 } }
I20260926 07:34:31.042651 11928 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7d071aff74644741aaa9e6d5140628c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:34:31.042713 11928 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7d071aff74644741aaa9e6d5140628c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d071aff74644741aaa9e6d5140628c8, State: Initialized, Role: FOLLOWER
I20260926 07:34:31.043267 11928 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d071aff74644741aaa9e6d5140628c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d071aff74644741aaa9e6d5140628c8" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 42313 } }
I20260926 07:34:31.043478 11928 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7d071aff74644741aaa9e6d5140628c8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:34:31.043555 11928 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7d071aff74644741aaa9e6d5140628c8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:34:31.043658 11928 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7d071aff74644741aaa9e6d5140628c8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:34:31.045154 11928 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7d071aff74644741aaa9e6d5140628c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d071aff74644741aaa9e6d5140628c8" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 42313 } }
I20260926 07:34:31.045586 11928 leader_election.cc:302] T 00000000000000000000000000000000 P 7d071aff74644741aaa9e6d5140628c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7d071aff74644741aaa9e6d5140628c8; no voters:
I20260926 07:34:31.045995 11928 leader_election.cc:288] T 00000000000000000000000000000000 P 7d071aff74644741aaa9e6d5140628c8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:34:31.046154 11938 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7d071aff74644741aaa9e6d5140628c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:34:31.046444 11938 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7d071aff74644741aaa9e6d5140628c8 [term 1 LEADER]: Becoming Leader. State: Replica: 7d071aff74644741aaa9e6d5140628c8, State: Running, Role: LEADER
I20260926 07:34:31.046870 11938 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7d071aff74644741aaa9e6d5140628c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d071aff74644741aaa9e6d5140628c8" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 42313 } }
I20260926 07:34:31.047467 11928 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d071aff74644741aaa9e6d5140628c8 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:34:31.048233 11938 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d071aff74644741aaa9e6d5140628c8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d071aff74644741aaa9e6d5140628c8. Latest consensus state: current_term: 1 leader_uuid: "7d071aff74644741aaa9e6d5140628c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d071aff74644741aaa9e6d5140628c8" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 42313 } } }
I20260926 07:34:31.048432 11938 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d071aff74644741aaa9e6d5140628c8 [sys.catalog]: This master's current role is: LEADER
I20260926 07:34:31.048892 11947 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:34:31.050566 11940 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d071aff74644741aaa9e6d5140628c8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7d071aff74644741aaa9e6d5140628c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d071aff74644741aaa9e6d5140628c8" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 42313 } } }
I20260926 07:34:31.050745 11940 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d071aff74644741aaa9e6d5140628c8 [sys.catalog]: This master's current role is: LEADER
I20260926 07:34:31.052567 11947 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:34:31.054949 11931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:31.044278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60723 (local address 127.22.53.254:42313)
0926 07:34:31.044433 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:31.044438 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:31.044751 (+ 313us) server_negotiation.cc:407] Connection header received
0926 07:34:31.044861 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:31.044864 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:31.044922 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:31.045017 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:31.045815 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:31.046625 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:31.047774 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:31.048116 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:31.048232 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:31.050324 (+ 2092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:31.050328 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:31.050331 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:31.050358 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:31.052144 (+ 1786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:31.052856 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:31.052860 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:31.052861 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:31.053032 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:31.053579 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:31.053583 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:31.053585 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:31.053834 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:31.053979 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:31.054679 (+ 700us) server_negotiation.cc:299] Negotiation successful
0926 07:34:31.054795 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 07:34:31.061188 11947 catalog_manager.cc:1347] Generated new cluster ID: 6f483c59f5ab4557af45d038ad007730
I20260926 07:34:31.061362 11947 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:34:31.077471 11947 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:34:31.078544 11947 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:34:31.085076 11947 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7d071aff74644741aaa9e6d5140628c8: Generated new TSK 0
I20260926 07:34:31.085906 11947 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:34:31.094874 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 11860
W20260926 07:34:31.101156 11845 connection.cc:535] client connection to 127.22.53.254:42313 recv error: Network error: failed to read from TLS socket (remote: 127.22.53.254:42313): Connection reset by peer (error 104)
2026-09-26T07:34:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1468 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407783529717-22743-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:34:31 dist-test-slave-wdjc krb5kdc[11959](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:34:31 dist-test-slave-wdjc krb5kdc[11959](info): set up 2 sockets
Sep 26 07:34:31 dist-test-slave-wdjc krb5kdc[11959](info): commencing operation
krb5kdc: starting...
W20260926 07:34:32.137488 22743 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407783529717-22743-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407783529717-22743-0/krb5kdc/test-user.keytab.
Sep 26 07:34:32 dist-test-slave-wdjc krb5kdc[11959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408072, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:34:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:34:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.53.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.53.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.53.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab.
Entry for principal kudu/127.22.53.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab.
I20260926 07:34:32.268455 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407783529717-22743-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407783529717-22743-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407783529717-22743-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.53.254:39821
--webserver_interface=127.22.53.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab
--principal=kudu/127.22.53.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:39993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.53.254:39821
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:34:32.390193 11972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:34:32.390532 11972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:34:32.390579 11972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:34:32.394857 11972 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:34:32.394939 11972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:34:32.394960 11972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:34:32.394977 11972 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:34:32.394992 11972 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:34:32.403672 11972 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:39993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407783529717-22743-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407783529717-22743-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.53.254:39821
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab
--principal=kudu/127.22.53.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.53.254:39821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.53.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11972
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407783529717-22743-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:34:32.405371 11972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:34:32.406948 11972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:34:32.418336 11977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:32.418380 11980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:34:32.418722 11972 server_base.cc:1034] running on GCE node
W20260926 07:34:32.418344 11978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:34:32.419565 11972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:34:32.421073 11972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:34:32.422267 11972 hybrid_clock.cc:648] HybridClock initialized: now 1790408072422236 us; error 53 us; skew 500 ppm
Sep 26 07:34:32 dist-test-slave-wdjc krb5kdc[11959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408072, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:34:32.425546 11972 init.cc:377] Logged in from keytab as kudu/127.22.53.254@KRBTEST.COM (short username kudu)
I20260926 07:34:32.426551 11972 webserver.cc:460] Webserver started at http://127.22.53.254:41757/ using document root <none> and password file <none>
I20260926 07:34:32.427117 11972 fs_manager.cc:359] Metadata directory not provided
I20260926 07:34:32.427189 11972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:34:32.427364 11972 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:34:32.429574 11972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407783529717-22743-0/minicluster-data/master-0/data/instance:
uuid: "1d28940b8e5a4932b08a37d8b44b5acd"
format_stamp: "Formatted at 2026-09-26 07:34:32 on dist-test-slave-wdjc"
server_key: "3f637cf9d60bf30887be1929382ca7cc"
server_key_iv: "38f66f61c1ef50360d987732e4e664dd"
server_key_version: "encryptionkey@0"
I20260926 07:34:32.430106 11972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407783529717-22743-0/minicluster-data/master-0/wal/instance:
uuid: "1d28940b8e5a4932b08a37d8b44b5acd"
format_stamp: "Formatted at 2026-09-26 07:34:32 on dist-test-slave-wdjc"
server_key: "3f637cf9d60bf30887be1929382ca7cc"
server_key_iv: "38f66f61c1ef50360d987732e4e664dd"
server_key_version: "encryptionkey@0"
I20260926 07:34:32.434240 11972 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 07:34:32.437258 11987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:32.438681 11972 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:34:32.438868 11972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407783529717-22743-0/minicluster-data/master-0/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407783529717-22743-0/minicluster-data/master-0/wal
uuid: "1d28940b8e5a4932b08a37d8b44b5acd"
format_stamp: "Formatted at 2026-09-26 07:34:32 on dist-test-slave-wdjc"
server_key: "3f637cf9d60bf30887be1929382ca7cc"
server_key_iv: "38f66f61c1ef50360d987732e4e664dd"
server_key_version: "encryptionkey@0"
I20260926 07:34:32.439009 11972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407783529717-22743-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407783529717-22743-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407783529717-22743-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:34:32.454973 11972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:34:32.458343 11972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:34:32.458570 11972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:34:32.466145 11972 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.254:39821
I20260926 07:34:32.466174 12039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.254:39821 every 8 connection(s)
I20260926 07:34:32.467289 11972 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
I20260926 07:34:32.470280 12040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:34:32.476068 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 11972
I20260926 07:34:32.476197 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407783529717-22743-0/minicluster-data/master-0/wal/instance
I20260926 07:34:32.476267 12040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d28940b8e5a4932b08a37d8b44b5acd: Bootstrap starting.
I20260926 07:34:32.476516 22743 external_mini_cluster.cc:1428] Setting key 154956d3fc21d922ad94330312068de6
I20260926 07:34:32.479080 12040 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d28940b8e5a4932b08a37d8b44b5acd: Neither blocks nor log segments found. Creating new log.
I20260926 07:34:32.480017 12040 log.cc:824] T 00000000000000000000000000000000 P 1d28940b8e5a4932b08a37d8b44b5acd: Log is configured to *not* fsync() on all Append() calls
I20260926 07:34:32.482290 12040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d28940b8e5a4932b08a37d8b44b5acd: No bootstrap required, opened a new log
I20260926 07:34:32.486915 12043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:32.478067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46531 (local address 127.22.53.254:39821)
0926 07:34:32.478545 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:32.478560 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:32.478609 (+ 49us) server_negotiation.cc:407] Connection header received
0926 07:34:32.479427 (+ 818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:32.479449 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:32.480158 (+ 709us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:32.480663 (+ 505us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:32.481120 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:32.482143 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:32.483365 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:32.483849 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:32.483978 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:32.485042 (+ 1064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:32.485047 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:32.485062 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:34:32.485492 (+ 430us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:32.485551 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:34:32.485576 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:32.485683 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:32.485897 (+ 214us) server_negotiation.cc:299] Negotiation successful
0926 07:34:32.486078 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":142,"threads_started":1}
I20260926 07:34:32.486862 12040 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d28940b8e5a4932b08a37d8b44b5acd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d28940b8e5a4932b08a37d8b44b5acd" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 39821 } }
I20260926 07:34:32.487200 12040 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1d28940b8e5a4932b08a37d8b44b5acd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:34:32.487282 12040 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d28940b8e5a4932b08a37d8b44b5acd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d28940b8e5a4932b08a37d8b44b5acd, State: Initialized, Role: FOLLOWER
I20260926 07:34:32.487775 12040 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d28940b8e5a4932b08a37d8b44b5acd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d28940b8e5a4932b08a37d8b44b5acd" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 39821 } }
I20260926 07:34:32.487946 12040 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1d28940b8e5a4932b08a37d8b44b5acd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:34:32.488057 12040 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d28940b8e5a4932b08a37d8b44b5acd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:34:32.488204 12040 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d28940b8e5a4932b08a37d8b44b5acd [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:34:32.489454 12040 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d28940b8e5a4932b08a37d8b44b5acd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d28940b8e5a4932b08a37d8b44b5acd" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 39821 } }
I20260926 07:34:32.489861 12040 leader_election.cc:302] T 00000000000000000000000000000000 P 1d28940b8e5a4932b08a37d8b44b5acd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1d28940b8e5a4932b08a37d8b44b5acd; no voters:
I20260926 07:34:32.490257 12040 leader_election.cc:288] T 00000000000000000000000000000000 P 1d28940b8e5a4932b08a37d8b44b5acd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:34:32.490384 12045 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d28940b8e5a4932b08a37d8b44b5acd [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:34:32.490429 22743 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:34:32.490666 12045 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d28940b8e5a4932b08a37d8b44b5acd [term 1 LEADER]: Becoming Leader. State: Replica: 1d28940b8e5a4932b08a37d8b44b5acd, State: Running, Role: LEADER
I20260926 07:34:32.491055 12045 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d28940b8e5a4932b08a37d8b44b5acd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d28940b8e5a4932b08a37d8b44b5acd" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 39821 } }
I20260926 07:34:32.491696 12040 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d28940b8e5a4932b08a37d8b44b5acd [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:34:32.492770 12047 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d28940b8e5a4932b08a37d8b44b5acd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1d28940b8e5a4932b08a37d8b44b5acd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d28940b8e5a4932b08a37d8b44b5acd" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 39821 } } }
I20260926 07:34:32.492945 12047 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d28940b8e5a4932b08a37d8b44b5acd [sys.catalog]: This master's current role is: LEADER
I20260926 07:34:32.493181 12047 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d28940b8e5a4932b08a37d8b44b5acd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d28940b8e5a4932b08a37d8b44b5acd. Latest consensus state: current_term: 1 leader_uuid: "1d28940b8e5a4932b08a37d8b44b5acd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d28940b8e5a4932b08a37d8b44b5acd" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 39821 } } }
I20260926 07:34:32.493255 12047 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d28940b8e5a4932b08a37d8b44b5acd [sys.catalog]: This master's current role is: LEADER
I20260926 07:34:32.493366 12059 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:34:32.497005 12059 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:34:32.499768 12043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:32.492768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47891 (local address 127.22.53.254:39821)
0926 07:34:32.492895 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:32.492900 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:32.493818 (+ 918us) server_negotiation.cc:407] Connection header received
0926 07:34:32.494036 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:32.494039 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:32.494087 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:32.494140 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:32.494579 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:32.495200 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:32.496508 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:32.497207 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:32.497910 (+ 703us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:32.498712 (+ 802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:32.498716 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:32.498719 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:34:32.498738 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:32.498776 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:34:32.498810 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:32.498922 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:32.499509 (+ 587us) server_negotiation.cc:299] Negotiation successful
0926 07:34:32.499559 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 07:34:32.504478 12059 catalog_manager.cc:1347] Generated new cluster ID: e334ebce806548c79f35fc3e424da6ec
I20260926 07:34:32.504586 12059 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:34:32.520499 12059 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:34:32.521823 12059 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:34:32.534466 12059 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1d28940b8e5a4932b08a37d8b44b5acd: Generated new TSK 0
I20260926 07:34:32.535431 12059 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:34:32.540392 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 11972
W20260926 07:34:32.546428 11957 connection.cc:535] client connection to 127.22.53.254:39821 recv error: Network error: failed to read from TLS socket (remote: 127.22.53.254:39821): Connection reset by peer (error 104)
2026-09-26T07:34:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1445 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407783529717-22743-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:34:32 dist-test-slave-wdjc krb5kdc[12071](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:34:32 dist-test-slave-wdjc krb5kdc[12071](info): set up 2 sockets
Sep 26 07:34:32 dist-test-slave-wdjc krb5kdc[12071](info): commencing operation
krb5kdc: starting...
W20260926 07:34:33.589008 22743 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407783529717-22743-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407783529717-22743-0/krb5kdc/test-user.keytab.
Sep 26 07:34:33 dist-test-slave-wdjc krb5kdc[12071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:34:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:34:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.53.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.53.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.53.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab.
Entry for principal kudu/127.22.53.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab.
I20260926 07:34:33.720271 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407783529717-22743-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407783529717-22743-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407783529717-22743-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.53.254:37651
--webserver_interface=127.22.53.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab
--principal=kudu/127.22.53.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:33855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.53.254:37651
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:34:33.843505 12084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:34:33.843823 12084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:34:33.843868 12084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:34:33.848120 12084 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:34:33.848206 12084 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:34:33.848232 12084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:34:33.848249 12084 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:34:33.848264 12084 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:34:33.853884 12084 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:33855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407783529717-22743-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407783529717-22743-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.53.254:37651
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab
--principal=kudu/127.22.53.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.53.254:37651
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.53.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12084
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407783529717-22743-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:34:33.855057 12084 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:34:33.856058 12084 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:34:33.865211 12089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:34:33.865311 12084 server_base.cc:1034] running on GCE node
W20260926 07:34:33.865242 12090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:33.865263 12092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:34:33.866160 12084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:34:33.867432 12084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:34:33.868639 12084 hybrid_clock.cc:648] HybridClock initialized: now 1790408073868621 us; error 41 us; skew 500 ppm
Sep 26 07:34:33 dist-test-slave-wdjc krb5kdc[12071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408073, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:34:33.871690 12084 init.cc:377] Logged in from keytab as kudu/127.22.53.254@KRBTEST.COM (short username kudu)
I20260926 07:34:33.872888 12084 webserver.cc:460] Webserver started at http://127.22.53.254:46417/ using document root <none> and password file <none>
I20260926 07:34:33.873508 12084 fs_manager.cc:359] Metadata directory not provided
I20260926 07:34:33.873577 12084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:34:33.873805 12084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:34:33.876071 12084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407783529717-22743-0/minicluster-data/master-0/data/instance:
uuid: "a1da33c061fc464394083ddbd3de3ef3"
format_stamp: "Formatted at 2026-09-26 07:34:33 on dist-test-slave-wdjc"
server_key: "1c57b54fea4952f0506068e08ed14e10"
server_key_iv: "60ec1e3beaed4ff8f1a5ba7e6aeadf90"
server_key_version: "encryptionkey@0"
I20260926 07:34:33.876761 12084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407783529717-22743-0/minicluster-data/master-0/wal/instance:
uuid: "a1da33c061fc464394083ddbd3de3ef3"
format_stamp: "Formatted at 2026-09-26 07:34:33 on dist-test-slave-wdjc"
server_key: "1c57b54fea4952f0506068e08ed14e10"
server_key_iv: "60ec1e3beaed4ff8f1a5ba7e6aeadf90"
server_key_version: "encryptionkey@0"
I20260926 07:34:33.880899 12084 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 07:34:33.883529 12099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:33.885149 12084 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:34:33.885349 12084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407783529717-22743-0/minicluster-data/master-0/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407783529717-22743-0/minicluster-data/master-0/wal
uuid: "a1da33c061fc464394083ddbd3de3ef3"
format_stamp: "Formatted at 2026-09-26 07:34:33 on dist-test-slave-wdjc"
server_key: "1c57b54fea4952f0506068e08ed14e10"
server_key_iv: "60ec1e3beaed4ff8f1a5ba7e6aeadf90"
server_key_version: "encryptionkey@0"
I20260926 07:34:33.885500 12084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407783529717-22743-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407783529717-22743-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407783529717-22743-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:34:33.890973 12084 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:34:33.894842 12084 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:34:33.895155 12084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:34:33.903370 12084 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.254:37651
I20260926 07:34:33.903455 12151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.254:37651 every 8 connection(s)
I20260926 07:34:33.904516 12084 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
I20260926 07:34:33.907811 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 12084
I20260926 07:34:33.907877 12152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:34:33.907979 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407783529717-22743-0/minicluster-data/master-0/wal/instance
I20260926 07:34:33.908286 22743 external_mini_cluster.cc:1428] Setting key 367d9f65c06378da7a4a42caa4fb643a
I20260926 07:34:33.914425 12155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:33.909886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51205 (local address 127.22.53.254:37651)
0926 07:34:33.910367 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:33.910383 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:33.910421 (+ 38us) server_negotiation.cc:407] Connection header received
0926 07:34:33.911383 (+ 962us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:33.911406 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:33.911732 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:33.912054 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:33.913101 (+ 1047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:33.913105 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:33.913119 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:34:33.913131 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:33.913191 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:34:33.913212 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:33.913333 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:33.913438 (+ 105us) server_negotiation.cc:299] Negotiation successful
0926 07:34:33.913624 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":167,"threads_started":1}
I20260926 07:34:33.915541 12152 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1da33c061fc464394083ddbd3de3ef3: Bootstrap starting.
I20260926 07:34:33.917709 22743 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:34:33.918309 12152 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a1da33c061fc464394083ddbd3de3ef3: Neither blocks nor log segments found. Creating new log.
I20260926 07:34:33.919351 12152 log.cc:824] T 00000000000000000000000000000000 P a1da33c061fc464394083ddbd3de3ef3: Log is configured to *not* fsync() on all Append() calls
I20260926 07:34:33.922171 12152 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1da33c061fc464394083ddbd3de3ef3: No bootstrap required, opened a new log
I20260926 07:34:33.922786 12155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:33.920473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45827 (local address 127.22.53.254:37651)
0926 07:34:33.920621 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:33.920626 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:33.920810 (+ 184us) server_negotiation.cc:407] Connection header received
0926 07:34:33.921345 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:33.921349 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:33.921393 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:33.921490 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:33.922368 (+ 878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:33.922371 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:33.922374 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:34:33.922386 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:33.922416 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:34:33.922429 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:33.922525 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:33.922622 (+ 97us) server_negotiation.cc:299] Negotiation successful
0926 07:34:33.922663 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 07:34:33.926250 12152 raft_consensus.cc:348] T 00000000000000000000000000000000 P a1da33c061fc464394083ddbd3de3ef3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1da33c061fc464394083ddbd3de3ef3" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 37651 } }
I20260926 07:34:33.926524 12152 raft_consensus.cc:374] T 00000000000000000000000000000000 P a1da33c061fc464394083ddbd3de3ef3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:34:33.926591 12152 raft_consensus.cc:729] T 00000000000000000000000000000000 P a1da33c061fc464394083ddbd3de3ef3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1da33c061fc464394083ddbd3de3ef3, State: Initialized, Role: FOLLOWER
I20260926 07:34:33.927129 12152 consensus_queue.cc:260] T 00000000000000000000000000000000 P a1da33c061fc464394083ddbd3de3ef3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1da33c061fc464394083ddbd3de3ef3" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 37651 } }
I20260926 07:34:33.927311 12152 raft_consensus.cc:388] T 00000000000000000000000000000000 P a1da33c061fc464394083ddbd3de3ef3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:34:33.927429 12152 raft_consensus.cc:482] T 00000000000000000000000000000000 P a1da33c061fc464394083ddbd3de3ef3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:34:33.927574 12152 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a1da33c061fc464394083ddbd3de3ef3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:34:33.928997 12152 raft_consensus.cc:504] T 00000000000000000000000000000000 P a1da33c061fc464394083ddbd3de3ef3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1da33c061fc464394083ddbd3de3ef3" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 37651 } }
I20260926 07:34:33.929422 12152 leader_election.cc:302] T 00000000000000000000000000000000 P a1da33c061fc464394083ddbd3de3ef3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a1da33c061fc464394083ddbd3de3ef3; no voters:
I20260926 07:34:33.929736 12152 leader_election.cc:288] T 00000000000000000000000000000000 P a1da33c061fc464394083ddbd3de3ef3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:34:33.929944 12162 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a1da33c061fc464394083ddbd3de3ef3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:34:33.930271 12162 raft_consensus.cc:686] T 00000000000000000000000000000000 P a1da33c061fc464394083ddbd3de3ef3 [term 1 LEADER]: Becoming Leader. State: Replica: a1da33c061fc464394083ddbd3de3ef3, State: Running, Role: LEADER
I20260926 07:34:33.930595 12162 consensus_queue.cc:237] T 00000000000000000000000000000000 P a1da33c061fc464394083ddbd3de3ef3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1da33c061fc464394083ddbd3de3ef3" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 37651 } }
I20260926 07:34:33.931104 12152 sys_catalog.cc:564] T 00000000000000000000000000000000 P a1da33c061fc464394083ddbd3de3ef3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:34:33.932350 12164 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1da33c061fc464394083ddbd3de3ef3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1da33c061fc464394083ddbd3de3ef3. Latest consensus state: current_term: 1 leader_uuid: "a1da33c061fc464394083ddbd3de3ef3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1da33c061fc464394083ddbd3de3ef3" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 37651 } } }
I20260926 07:34:33.932490 12164 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1da33c061fc464394083ddbd3de3ef3 [sys.catalog]: This master's current role is: LEADER
I20260926 07:34:33.932839 12163 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1da33c061fc464394083ddbd3de3ef3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a1da33c061fc464394083ddbd3de3ef3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1da33c061fc464394083ddbd3de3ef3" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 37651 } } }
I20260926 07:34:33.932919 12171 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:34:33.932955 12163 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1da33c061fc464394083ddbd3de3ef3 [sys.catalog]: This master's current role is: LEADER
I20260926 07:34:33.936605 12171 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:34:33.943034 12171 catalog_manager.cc:1347] Generated new cluster ID: fcfa42241f484a2ebe7393274f27486b
I20260926 07:34:33.943148 12171 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:34:33.964401 12171 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:34:33.964861 12171 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:34:33.969776 12171 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a1da33c061fc464394083ddbd3de3ef3: Generated new TSK 0
I20260926 07:34:33.970575 12171 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:34:33.977638 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 12084
2026-09-26T07:34:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1436 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407783529717-22743-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:34:34 dist-test-slave-wdjc krb5kdc[12183](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:34:34 dist-test-slave-wdjc krb5kdc[12183](info): set up 2 sockets
Sep 26 07:34:34 dist-test-slave-wdjc krb5kdc[12183](info): commencing operation
krb5kdc: starting...
W20260926 07:34:35.049036 22743 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.007s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407783529717-22743-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407783529717-22743-0/krb5kdc/test-user.keytab.
Sep 26 07:34:35 dist-test-slave-wdjc krb5kdc[12183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408075, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:34:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:34:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.53.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.53.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.53.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab.
Entry for principal kudu/127.22.53.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab.
I20260926 07:34:35.176216 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407783529717-22743-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407783529717-22743-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407783529717-22743-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.53.254:41433
--webserver_interface=127.22.53.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab
--principal=kudu/127.22.53.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:33817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.53.254:41433
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:34:35.296409 12196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:34:35.296818 12196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:34:35.296913 12196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:34:35.301244 12196 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:34:35.301380 12196 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:34:35.301420 12196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:34:35.301460 12196 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:34:35.301492 12196 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:34:35.307754 12196 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:33817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407783529717-22743-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407783529717-22743-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.53.254:41433
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab
--principal=kudu/127.22.53.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.53.254:41433
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.53.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12196
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407783529717-22743-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:34:35.309126 12196 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:34:35.310324 12196 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:34:35.319963 12204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:35.319957 12202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:35.319986 12201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:34:35.320129 12196 server_base.cc:1034] running on GCE node
I20260926 07:34:35.321005 12196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:34:35.322265 12196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:34:35.323503 12196 hybrid_clock.cc:648] HybridClock initialized: now 1790408075323481 us; error 51 us; skew 500 ppm
Sep 26 07:34:35 dist-test-slave-wdjc krb5kdc[12183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408075, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:34:35.326932 12196 init.cc:377] Logged in from keytab as kudu/127.22.53.254@KRBTEST.COM (short username kudu)
I20260926 07:34:35.328027 12196 webserver.cc:460] Webserver started at http://127.22.53.254:36339/ using document root <none> and password file <none>
I20260926 07:34:35.328780 12196 fs_manager.cc:359] Metadata directory not provided
I20260926 07:34:35.329021 12196 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:34:35.329315 12196 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:34:35.331933 12196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407783529717-22743-0/minicluster-data/master-0/data/instance:
uuid: "ac6ae8f51ced4722b33202ebdba33b42"
format_stamp: "Formatted at 2026-09-26 07:34:35 on dist-test-slave-wdjc"
server_key: "a9df47243eeb51cbeac9410e8609e3ce"
server_key_iv: "31587eae5011c747ac6371825da038f9"
server_key_version: "encryptionkey@0"
I20260926 07:34:35.332688 12196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407783529717-22743-0/minicluster-data/master-0/wal/instance:
uuid: "ac6ae8f51ced4722b33202ebdba33b42"
format_stamp: "Formatted at 2026-09-26 07:34:35 on dist-test-slave-wdjc"
server_key: "a9df47243eeb51cbeac9410e8609e3ce"
server_key_iv: "31587eae5011c747ac6371825da038f9"
server_key_version: "encryptionkey@0"
I20260926 07:34:35.337081 12196 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 07:34:35.340072 12211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:35.341581 12196 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:34:35.341764 12196 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407783529717-22743-0/minicluster-data/master-0/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407783529717-22743-0/minicluster-data/master-0/wal
uuid: "ac6ae8f51ced4722b33202ebdba33b42"
format_stamp: "Formatted at 2026-09-26 07:34:35 on dist-test-slave-wdjc"
server_key: "a9df47243eeb51cbeac9410e8609e3ce"
server_key_iv: "31587eae5011c747ac6371825da038f9"
server_key_version: "encryptionkey@0"
I20260926 07:34:35.341935 12196 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407783529717-22743-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407783529717-22743-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407783529717-22743-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:34:35.367486 12196 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:34:35.370922 12196 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:34:35.371167 12196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:34:35.378861 12196 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.254:41433
I20260926 07:34:35.378876 12263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.254:41433 every 8 connection(s)
I20260926 07:34:35.379906 12196 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
I20260926 07:34:35.383291 12264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:34:35.384464 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 12196
I20260926 07:34:35.384661 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407783529717-22743-0/minicluster-data/master-0/wal/instance
I20260926 07:34:35.384958 22743 external_mini_cluster.cc:1428] Setting key 83f56d0e14c17be1c0e36b24ac23c9e4
I20260926 07:34:35.390528 12264 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac6ae8f51ced4722b33202ebdba33b42: Bootstrap starting.
I20260926 07:34:35.393769 12264 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac6ae8f51ced4722b33202ebdba33b42: Neither blocks nor log segments found. Creating new log.
I20260926 07:34:35.394867 12264 log.cc:824] T 00000000000000000000000000000000 P ac6ae8f51ced4722b33202ebdba33b42: Log is configured to *not* fsync() on all Append() calls
Sep 26 07:34:35 dist-test-slave-wdjc krb5kdc[12183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408075, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.53.254@KRBTEST.COM
I20260926 07:34:35.397326 12264 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac6ae8f51ced4722b33202ebdba33b42: No bootstrap required, opened a new log
I20260926 07:34:35.401222 12267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:35.386550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37627 (local address 127.22.53.254:41433)
0926 07:34:35.387108 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:35.387124 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:35.387168 (+ 44us) server_negotiation.cc:407] Connection header received
0926 07:34:35.387847 (+ 679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:35.387866 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:35.388237 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:35.388669 (+ 432us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:35.389550 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:35.390584 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:35.391971 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:35.392480 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:35.392747 (+ 267us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:35.395993 (+ 3246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:35.396000 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:35.396017 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:35.396049 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:35.398015 (+ 1966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:35.398546 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:35.398551 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:35.398556 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:35.398681 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:35.398997 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:35.399005 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:35.399006 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:35.399452 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:35.399680 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:35.399993 (+ 313us) server_negotiation.cc:299] Negotiation successful
0926 07:34:35.400329 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":170,"threads_started":1}
I20260926 07:34:35.401911 12264 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac6ae8f51ced4722b33202ebdba33b42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac6ae8f51ced4722b33202ebdba33b42" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 41433 } }
I20260926 07:34:35.402213 12264 raft_consensus.cc:374] T 00000000000000000000000000000000 P ac6ae8f51ced4722b33202ebdba33b42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:34:35.402263 12264 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac6ae8f51ced4722b33202ebdba33b42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac6ae8f51ced4722b33202ebdba33b42, State: Initialized, Role: FOLLOWER
I20260926 07:34:35.402750 12264 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac6ae8f51ced4722b33202ebdba33b42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac6ae8f51ced4722b33202ebdba33b42" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 41433 } }
I20260926 07:34:35.402918 12264 raft_consensus.cc:388] T 00000000000000000000000000000000 P ac6ae8f51ced4722b33202ebdba33b42 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:34:35.403007 12264 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac6ae8f51ced4722b33202ebdba33b42 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:34:35.403134 12264 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac6ae8f51ced4722b33202ebdba33b42 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:34:35.404728 12264 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac6ae8f51ced4722b33202ebdba33b42 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac6ae8f51ced4722b33202ebdba33b42" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 41433 } }
I20260926 07:34:35.405169 22743 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:34:35.405195 12264 leader_election.cc:302] T 00000000000000000000000000000000 P ac6ae8f51ced4722b33202ebdba33b42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac6ae8f51ced4722b33202ebdba33b42; no voters:
I20260926 07:34:35.405555 12264 leader_election.cc:288] T 00000000000000000000000000000000 P ac6ae8f51ced4722b33202ebdba33b42 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:34:35.405699 12269 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac6ae8f51ced4722b33202ebdba33b42 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:34:35.406363 12269 raft_consensus.cc:686] T 00000000000000000000000000000000 P ac6ae8f51ced4722b33202ebdba33b42 [term 1 LEADER]: Becoming Leader. State: Replica: ac6ae8f51ced4722b33202ebdba33b42, State: Running, Role: LEADER
I20260926 07:34:35.406891 12269 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac6ae8f51ced4722b33202ebdba33b42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac6ae8f51ced4722b33202ebdba33b42" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 41433 } }
I20260926 07:34:35.407162 12264 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac6ae8f51ced4722b33202ebdba33b42 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:34:35.408803 12271 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac6ae8f51ced4722b33202ebdba33b42 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ac6ae8f51ced4722b33202ebdba33b42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac6ae8f51ced4722b33202ebdba33b42" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 41433 } } }
I20260926 07:34:35.408949 12271 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac6ae8f51ced4722b33202ebdba33b42 [sys.catalog]: This master's current role is: LEADER
I20260926 07:34:35.409327 12270 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac6ae8f51ced4722b33202ebdba33b42 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac6ae8f51ced4722b33202ebdba33b42. Latest consensus state: current_term: 1 leader_uuid: "ac6ae8f51ced4722b33202ebdba33b42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac6ae8f51ced4722b33202ebdba33b42" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 41433 } } }
I20260926 07:34:35.409374 12285 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:34:35.409420 12270 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac6ae8f51ced4722b33202ebdba33b42 [sys.catalog]: This master's current role is: LEADER
I20260926 07:34:35.412822 12285 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:34:35.420890 12267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:35.410618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36985 (local address 127.22.53.254:41433)
0926 07:34:35.410754 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:35.410760 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:35.411220 (+ 460us) server_negotiation.cc:407] Connection header received
0926 07:34:35.411455 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:35.411458 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:35.411519 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:35.411616 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:35.412402 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:35.413167 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:35.414089 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:35.414373 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:35.414494 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:35.416321 (+ 1827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:35.416325 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:35.416328 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:35.416364 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:35.418334 (+ 1970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:35.418936 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:35.418942 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:35.418945 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:35.419055 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:35.419422 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:35.419425 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:35.419427 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:35.419693 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:35.419846 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:35.420180 (+ 334us) server_negotiation.cc:299] Negotiation successful
0926 07:34:35.420382 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 07:34:35.420403 12285 catalog_manager.cc:1347] Generated new cluster ID: ee337bf8f35940f7af1349f1bcc74b4a
I20260926 07:34:35.421281 12285 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:34:35.434229 12285 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:34:35.435627 12285 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:34:35.446596 12285 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ac6ae8f51ced4722b33202ebdba33b42: Generated new TSK 0
I20260926 07:34:35.447582 12285 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:34:35.460197 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 12196
2026-09-26T07:34:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1482 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407783529717-22743-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:34:35 dist-test-slave-wdjc krb5kdc[12295](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:34:35 dist-test-slave-wdjc krb5kdc[12295](info): set up 2 sockets
Sep 26 07:34:35 dist-test-slave-wdjc krb5kdc[12295](info): commencing operation
krb5kdc: starting...
W20260926 07:34:36.525336 22743 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407783529717-22743-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407783529717-22743-0/krb5kdc/test-user.keytab.
Sep 26 07:34:36 dist-test-slave-wdjc krb5kdc[12295](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408076, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:34:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:34:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.53.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.53.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.53.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab.
Entry for principal kudu/127.22.53.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab.
I20260926 07:34:36.661360 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407783529717-22743-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407783529717-22743-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407783529717-22743-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.53.254:37391
--webserver_interface=127.22.53.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab
--principal=kudu/127.22.53.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:38677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.53.254:37391
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:34:36.778465 12308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:34:36.778815 12308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:34:36.778898 12308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:34:36.783368 12308 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:34:36.783481 12308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:34:36.783517 12308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:34:36.783560 12308 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:34:36.783628 12308 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:34:36.789484 12308 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:38677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407783529717-22743-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407783529717-22743-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.53.254:37391
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab
--principal=kudu/127.22.53.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.53.254:37391
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.53.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12308
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407783529717-22743-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:34:36.790741 12308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:34:36.791824 12308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:34:36.800937 12314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:36.800937 12313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:36.800938 12316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:34:36.801419 12308 server_base.cc:1034] running on GCE node
I20260926 07:34:36.802084 12308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:34:36.803292 12308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:34:36.804513 12308 hybrid_clock.cc:648] HybridClock initialized: now 1790408076804494 us; error 36 us; skew 500 ppm
Sep 26 07:34:36 dist-test-slave-wdjc krb5kdc[12295](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408076, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:34:36.812460 12308 init.cc:377] Logged in from keytab as kudu/127.22.53.254@KRBTEST.COM (short username kudu)
I20260926 07:34:36.813617 12308 webserver.cc:460] Webserver started at http://127.22.53.254:37169/ using document root <none> and password file <none>
I20260926 07:34:36.814263 12308 fs_manager.cc:359] Metadata directory not provided
I20260926 07:34:36.814364 12308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:34:36.814612 12308 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:34:36.816936 12308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407783529717-22743-0/minicluster-data/master-0/data/instance:
uuid: "b8651db0f8f44f3badbe808b7a027bae"
format_stamp: "Formatted at 2026-09-26 07:34:36 on dist-test-slave-wdjc"
server_key: "26ee0e4341abd09493c25f3a4c27a185"
server_key_iv: "f9f24975511bbcc570fc05d815e4d3f4"
server_key_version: "encryptionkey@0"
I20260926 07:34:36.817584 12308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407783529717-22743-0/minicluster-data/master-0/wal/instance:
uuid: "b8651db0f8f44f3badbe808b7a027bae"
format_stamp: "Formatted at 2026-09-26 07:34:36 on dist-test-slave-wdjc"
server_key: "26ee0e4341abd09493c25f3a4c27a185"
server_key_iv: "f9f24975511bbcc570fc05d815e4d3f4"
server_key_version: "encryptionkey@0"
I20260926 07:34:36.821776 12308 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 07:34:36.824713 12323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:36.826143 12308 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 07:34:36.826371 12308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407783529717-22743-0/minicluster-data/master-0/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407783529717-22743-0/minicluster-data/master-0/wal
uuid: "b8651db0f8f44f3badbe808b7a027bae"
format_stamp: "Formatted at 2026-09-26 07:34:36 on dist-test-slave-wdjc"
server_key: "26ee0e4341abd09493c25f3a4c27a185"
server_key_iv: "f9f24975511bbcc570fc05d815e4d3f4"
server_key_version: "encryptionkey@0"
I20260926 07:34:36.826524 12308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407783529717-22743-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407783529717-22743-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407783529717-22743-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:34:36.845402 12308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:34:36.848829 12308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:34:36.849076 12308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:34:36.856696 12308 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.254:37391
I20260926 07:34:36.856711 12375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.254:37391 every 8 connection(s)
I20260926 07:34:36.857826 12308 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
I20260926 07:34:36.858846 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 12308
I20260926 07:34:36.858982 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407783529717-22743-0/minicluster-data/master-0/wal/instance
I20260926 07:34:36.859285 22743 external_mini_cluster.cc:1428] Setting key 0cc424696b81fabeb9e87510660d8baf
I20260926 07:34:36.861444 12376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:34:36.868376 12376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8651db0f8f44f3badbe808b7a027bae: Bootstrap starting.
I20260926 07:34:36.869814 12379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:36.860828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43465 (local address 127.22.53.254:37391)
0926 07:34:36.861246 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:36.861264 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:36.861330 (+ 66us) server_negotiation.cc:407] Connection header received
0926 07:34:36.862396 (+ 1066us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:36.862427 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:36.862934 (+ 507us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:36.863357 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:36.863833 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:36.865041 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:36.866300 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:36.866828 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:36.866969 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:36.867967 (+ 998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:36.867972 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:36.867986 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:34:36.868401 (+ 415us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:36.868470 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:34:36.868499 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:36.868675 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:36.868797 (+ 122us) server_negotiation.cc:299] Negotiation successful
0926 07:34:36.869049 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":135,"threads_started":1}
I20260926 07:34:36.870944 12376 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b8651db0f8f44f3badbe808b7a027bae: Neither blocks nor log segments found. Creating new log.
I20260926 07:34:36.871728 12376 log.cc:824] T 00000000000000000000000000000000 P b8651db0f8f44f3badbe808b7a027bae: Log is configured to *not* fsync() on all Append() calls
I20260926 07:34:36.872991 22743 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:34:36.874507 12376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8651db0f8f44f3badbe808b7a027bae: No bootstrap required, opened a new log
I20260926 07:34:36.879519 12376 raft_consensus.cc:348] T 00000000000000000000000000000000 P b8651db0f8f44f3badbe808b7a027bae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8651db0f8f44f3badbe808b7a027bae" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 37391 } }
I20260926 07:34:36.879905 12376 raft_consensus.cc:374] T 00000000000000000000000000000000 P b8651db0f8f44f3badbe808b7a027bae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:34:36.880034 12376 raft_consensus.cc:729] T 00000000000000000000000000000000 P b8651db0f8f44f3badbe808b7a027bae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8651db0f8f44f3badbe808b7a027bae, State: Initialized, Role: FOLLOWER
I20260926 07:34:36.880085 12379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:36.875512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39019 (local address 127.22.53.254:37391)
0926 07:34:36.875649 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:36.875654 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:36.875832 (+ 178us) server_negotiation.cc:407] Connection header received
0926 07:34:36.875929 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:36.875932 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:36.875976 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:36.876051 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:36.876385 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:36.877019 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:36.878299 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:36.878544 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:36.878631 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:36.879566 (+ 935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:36.879569 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:36.879571 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:34:36.879584 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:36.879611 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:34:36.879621 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:36.879704 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:36.879822 (+ 118us) server_negotiation.cc:299] Negotiation successful
0926 07:34:36.879908 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 07:34:36.880640 12376 consensus_queue.cc:260] T 00000000000000000000000000000000 P b8651db0f8f44f3badbe808b7a027bae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8651db0f8f44f3badbe808b7a027bae" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 37391 } }
I20260926 07:34:36.880802 12376 raft_consensus.cc:388] T 00000000000000000000000000000000 P b8651db0f8f44f3badbe808b7a027bae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:34:36.880890 12376 raft_consensus.cc:482] T 00000000000000000000000000000000 P b8651db0f8f44f3badbe808b7a027bae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:34:36.881016 12376 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b8651db0f8f44f3badbe808b7a027bae [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:34:36.882059 12376 raft_consensus.cc:504] T 00000000000000000000000000000000 P b8651db0f8f44f3badbe808b7a027bae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8651db0f8f44f3badbe808b7a027bae" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 37391 } }
I20260926 07:34:36.882443 12376 leader_election.cc:302] T 00000000000000000000000000000000 P b8651db0f8f44f3badbe808b7a027bae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b8651db0f8f44f3badbe808b7a027bae; no voters:
I20260926 07:34:36.882792 12376 leader_election.cc:288] T 00000000000000000000000000000000 P b8651db0f8f44f3badbe808b7a027bae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:34:36.882926 12386 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b8651db0f8f44f3badbe808b7a027bae [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:34:36.883173 12386 raft_consensus.cc:686] T 00000000000000000000000000000000 P b8651db0f8f44f3badbe808b7a027bae [term 1 LEADER]: Becoming Leader. State: Replica: b8651db0f8f44f3badbe808b7a027bae, State: Running, Role: LEADER
I20260926 07:34:36.883540 12386 consensus_queue.cc:237] T 00000000000000000000000000000000 P b8651db0f8f44f3badbe808b7a027bae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8651db0f8f44f3badbe808b7a027bae" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 37391 } }
I20260926 07:34:36.884150 12376 sys_catalog.cc:564] T 00000000000000000000000000000000 P b8651db0f8f44f3badbe808b7a027bae [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:34:36.885541 12388 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8651db0f8f44f3badbe808b7a027bae [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8651db0f8f44f3badbe808b7a027bae. Latest consensus state: current_term: 1 leader_uuid: "b8651db0f8f44f3badbe808b7a027bae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8651db0f8f44f3badbe808b7a027bae" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 37391 } } }
I20260926 07:34:36.885546 12387 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8651db0f8f44f3badbe808b7a027bae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b8651db0f8f44f3badbe808b7a027bae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8651db0f8f44f3badbe808b7a027bae" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 37391 } } }
I20260926 07:34:36.885726 12388 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8651db0f8f44f3badbe808b7a027bae [sys.catalog]: This master's current role is: LEADER
I20260926 07:34:36.885749 12387 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8651db0f8f44f3badbe808b7a027bae [sys.catalog]: This master's current role is: LEADER
I20260926 07:34:36.886116 12396 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:34:36.889243 12396 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:34:36.895416 12396 catalog_manager.cc:1347] Generated new cluster ID: 6b13f2a986f34c4c9a1fa1ef4e1bde3a
I20260926 07:34:36.895514 12396 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:34:36.907992 12396 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:34:36.909104 12396 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:34:36.919072 12396 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b8651db0f8f44f3badbe808b7a027bae: Generated new TSK 0
I20260926 07:34:36.919914 12396 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:34:36.929765 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 12308
2026-09-26T07:34:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1469 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407783529717-22743-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:34:36 dist-test-slave-wdjc krb5kdc[12407](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:34:36 dist-test-slave-wdjc krb5kdc[12407](info): set up 2 sockets
Sep 26 07:34:36 dist-test-slave-wdjc krb5kdc[12407](info): commencing operation
krb5kdc: starting...
W20260926 07:34:37.985014 22743 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407783529717-22743-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407783529717-22743-0/krb5kdc/test-user.keytab.
Sep 26 07:34:38 dist-test-slave-wdjc krb5kdc[12407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408078, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:34:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:34:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.53.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.53.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.53.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab.
Entry for principal kudu/127.22.53.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab.
I20260926 07:34:38.116513 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407783529717-22743-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407783529717-22743-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407783529717-22743-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.53.254:46111
--webserver_interface=127.22.53.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab
--principal=kudu/127.22.53.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:34147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.53.254:46111
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:34:38.238507 12420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:34:38.238828 12420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:34:38.238873 12420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:34:38.243175 12420 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:34:38.243266 12420 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:34:38.243286 12420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:34:38.243304 12420 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:34:38.243320 12420 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:34:38.249715 12420 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:34147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407783529717-22743-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407783529717-22743-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.53.254:46111
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab
--principal=kudu/127.22.53.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.53.254:46111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.53.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12420
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407783529717-22743-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:34:38.251236 12420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:34:38.252302 12420 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:34:38.261528 12425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:34:38.261582 12420 server_base.cc:1034] running on GCE node
W20260926 07:34:38.261534 12426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:38.261528 12428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:34:38.262560 12420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:34:38.263926 12420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:34:38.265161 12420 hybrid_clock.cc:648] HybridClock initialized: now 1790408078265128 us; error 49 us; skew 500 ppm
Sep 26 07:34:38 dist-test-slave-wdjc krb5kdc[12407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408078, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:34:38.268497 12420 init.cc:377] Logged in from keytab as kudu/127.22.53.254@KRBTEST.COM (short username kudu)
I20260926 07:34:38.269658 12420 webserver.cc:460] Webserver started at http://127.22.53.254:35303/ using document root <none> and password file <none>
I20260926 07:34:38.270341 12420 fs_manager.cc:359] Metadata directory not provided
I20260926 07:34:38.270421 12420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:34:38.270679 12420 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:34:38.273222 12420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407783529717-22743-0/minicluster-data/master-0/data/instance:
uuid: "4df98f18fbc247f5b614e99272962c1b"
format_stamp: "Formatted at 2026-09-26 07:34:38 on dist-test-slave-wdjc"
server_key: "3beb4067d0d91f16daf8ea00bac18ef7"
server_key_iv: "e6059a5a6520abdc3c5379e33fcb17b2"
server_key_version: "encryptionkey@0"
I20260926 07:34:38.273881 12420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407783529717-22743-0/minicluster-data/master-0/wal/instance:
uuid: "4df98f18fbc247f5b614e99272962c1b"
format_stamp: "Formatted at 2026-09-26 07:34:38 on dist-test-slave-wdjc"
server_key: "3beb4067d0d91f16daf8ea00bac18ef7"
server_key_iv: "e6059a5a6520abdc3c5379e33fcb17b2"
server_key_version: "encryptionkey@0"
I20260926 07:34:38.278398 12420 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 07:34:38.281497 12435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:38.283077 12420 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:34:38.283270 12420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407783529717-22743-0/minicluster-data/master-0/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407783529717-22743-0/minicluster-data/master-0/wal
uuid: "4df98f18fbc247f5b614e99272962c1b"
format_stamp: "Formatted at 2026-09-26 07:34:38 on dist-test-slave-wdjc"
server_key: "3beb4067d0d91f16daf8ea00bac18ef7"
server_key_iv: "e6059a5a6520abdc3c5379e33fcb17b2"
server_key_version: "encryptionkey@0"
I20260926 07:34:38.283427 12420 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407783529717-22743-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407783529717-22743-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407783529717-22743-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:34:38.288717 12420 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:34:38.292030 12420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:34:38.292295 12420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:34:38.300230 12420 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.254:46111
I20260926 07:34:38.300258 12487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.254:46111 every 8 connection(s)
I20260926 07:34:38.301893 12420 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
I20260926 07:34:38.304306 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 12420
I20260926 07:34:38.304437 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407783529717-22743-0/minicluster-data/master-0/wal/instance
I20260926 07:34:38.304772 22743 external_mini_cluster.cc:1428] Setting key 11c16a4dfaf3353cf0d2c02a90eba4dd
I20260926 07:34:38.305467 12488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:34:38.311426 12491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:38.306333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53477 (local address 127.22.53.254:46111)
0926 07:34:38.306910 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:38.306926 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:38.307141 (+ 215us) server_negotiation.cc:407] Connection header received
0926 07:34:38.308036 (+ 895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:38.308056 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:38.308386 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:38.308844 (+ 458us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:38.309946 (+ 1102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:38.309952 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:38.309966 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:34:38.309978 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:38.310041 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:34:38.310065 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:38.310201 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:38.310352 (+ 151us) server_negotiation.cc:299] Negotiation successful
0926 07:34:38.310567 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":179,"threads_started":1}
I20260926 07:34:38.313666 12488 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4df98f18fbc247f5b614e99272962c1b: Bootstrap starting.
I20260926 07:34:38.314791 22743 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:34:38.316780 12488 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4df98f18fbc247f5b614e99272962c1b: Neither blocks nor log segments found. Creating new log.
I20260926 07:34:38.317759 12488 log.cc:824] T 00000000000000000000000000000000 P 4df98f18fbc247f5b614e99272962c1b: Log is configured to *not* fsync() on all Append() calls
I20260926 07:34:38.319365 12491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:38.317397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56827 (local address 127.22.53.254:46111)
0926 07:34:38.317544 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:38.317549 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:38.317747 (+ 198us) server_negotiation.cc:407] Connection header received
0926 07:34:38.317930 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:38.317939 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:38.317989 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:38.318056 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:38.318911 (+ 855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:38.318914 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:38.318917 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:34:38.318927 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:38.318954 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:34:38.318965 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:38.319038 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:38.319135 (+ 97us) server_negotiation.cc:299] Negotiation successful
0926 07:34:38.319171 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 07:34:38.320421 12488 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4df98f18fbc247f5b614e99272962c1b: No bootstrap required, opened a new log
I20260926 07:34:38.324162 12488 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4df98f18fbc247f5b614e99272962c1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4df98f18fbc247f5b614e99272962c1b" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 46111 } }
I20260926 07:34:38.324514 12488 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4df98f18fbc247f5b614e99272962c1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:34:38.324608 12488 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4df98f18fbc247f5b614e99272962c1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4df98f18fbc247f5b614e99272962c1b, State: Initialized, Role: FOLLOWER
I20260926 07:34:38.325101 12488 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4df98f18fbc247f5b614e99272962c1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4df98f18fbc247f5b614e99272962c1b" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 46111 } }
I20260926 07:34:38.325279 12488 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4df98f18fbc247f5b614e99272962c1b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:34:38.325376 12488 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4df98f18fbc247f5b614e99272962c1b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:34:38.325505 12488 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4df98f18fbc247f5b614e99272962c1b [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:34:38.326910 12488 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4df98f18fbc247f5b614e99272962c1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4df98f18fbc247f5b614e99272962c1b" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 46111 } }
I20260926 07:34:38.327325 12488 leader_election.cc:302] T 00000000000000000000000000000000 P 4df98f18fbc247f5b614e99272962c1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4df98f18fbc247f5b614e99272962c1b; no voters:
I20260926 07:34:38.327689 12488 leader_election.cc:288] T 00000000000000000000000000000000 P 4df98f18fbc247f5b614e99272962c1b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:34:38.327894 12498 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4df98f18fbc247f5b614e99272962c1b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:34:38.328311 12498 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4df98f18fbc247f5b614e99272962c1b [term 1 LEADER]: Becoming Leader. State: Replica: 4df98f18fbc247f5b614e99272962c1b, State: Running, Role: LEADER
I20260926 07:34:38.328727 12498 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4df98f18fbc247f5b614e99272962c1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4df98f18fbc247f5b614e99272962c1b" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 46111 } }
I20260926 07:34:38.329208 12488 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4df98f18fbc247f5b614e99272962c1b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:34:38.330550 12499 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4df98f18fbc247f5b614e99272962c1b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4df98f18fbc247f5b614e99272962c1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4df98f18fbc247f5b614e99272962c1b" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 46111 } } }
I20260926 07:34:38.330729 12499 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4df98f18fbc247f5b614e99272962c1b [sys.catalog]: This master's current role is: LEADER
I20260926 07:34:38.330945 12500 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4df98f18fbc247f5b614e99272962c1b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4df98f18fbc247f5b614e99272962c1b. Latest consensus state: current_term: 1 leader_uuid: "4df98f18fbc247f5b614e99272962c1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4df98f18fbc247f5b614e99272962c1b" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 46111 } } }
I20260926 07:34:38.331084 12500 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4df98f18fbc247f5b614e99272962c1b [sys.catalog]: This master's current role is: LEADER
I20260926 07:34:38.332201 12509 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:34:38.335114 12509 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:34:38.341341 12509 catalog_manager.cc:1347] Generated new cluster ID: 7d77b9f97878455580c359cbdac3755d
I20260926 07:34:38.341485 12509 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:34:38.361649 12509 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:34:38.362116 12509 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:34:38.368575 12509 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4df98f18fbc247f5b614e99272962c1b: Generated new TSK 0
I20260926 07:34:38.369405 12509 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:34:38.377578 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 12420
2026-09-26T07:34:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1448 ms)
[----------] 18 tests from AuthTokenIssuingTest (26266 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407783529717-22743-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:34:38 dist-test-slave-wdjc krb5kdc[12519](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:34:38 dist-test-slave-wdjc krb5kdc[12519](info): set up 2 sockets
Sep 26 07:34:38 dist-test-slave-wdjc krb5kdc[12519](info): commencing operation
krb5kdc: starting...
W20260926 07:34:39.433348 22743 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407783529717-22743-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407783529717-22743-0/krb5kdc/test-user.keytab.
Sep 26 07:34:39 dist-test-slave-wdjc krb5kdc[12519](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408079, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:34:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:34:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.53.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.53.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.53.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab.
Entry for principal kudu/127.22.53.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab.
I20260926 07:34:39.563501 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407783529717-22743-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407783529717-22743-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407783529717-22743-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.53.254:46639
--webserver_interface=127.22.53.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab
--principal=kudu/127.22.53.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:41329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.53.254:46639,127.22.53.253:40235,127.22.53.252:39951
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:34:39.680037 12532 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 07:34:39.680352 12532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:34:39.680379 12532 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:34:39.680462 12532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:34:39.684804 12532 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:34:39.684888 12532 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:34:39.684908 12532 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:34:39.684926 12532 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:34:39.684940 12532 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:34:39.690717 12532 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:41329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407783529717-22743-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407783529717-22743-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.53.254:46639,127.22.53.253:40235,127.22.53.252:39951
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab
--principal=kudu/127.22.53.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.53.254:46639
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.53.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12532
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407783529717-22743-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:34:39.691910 12532 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:34:39.692996 12532 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:34:39.701855 12537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:39.701866 12540 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:34:39.701893 12532 server_base.cc:1034] running on GCE node
W20260926 07:34:39.701920 12538 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:34:39.702664 12532 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:34:39.703966 12532 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:34:39.705178 12532 hybrid_clock.cc:648] HybridClock initialized: now 1790408079705147 us; error 50 us; skew 500 ppm
Sep 26 07:34:39 dist-test-slave-wdjc krb5kdc[12519](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408079, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:34:39.708292 12532 init.cc:377] Logged in from keytab as kudu/127.22.53.254@KRBTEST.COM (short username kudu)
I20260926 07:34:39.709285 12532 webserver.cc:460] Webserver started at http://127.22.53.254:39053/ using document root <none> and password file <none>
I20260926 07:34:39.709826 12532 fs_manager.cc:359] Metadata directory not provided
I20260926 07:34:39.709898 12532 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:34:39.710078 12532 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:34:39.712204 12532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407783529717-22743-0/minicluster-data/master-0/data/instance:
uuid: "4f07f342df5d4f609220ec3f3993d24b"
format_stamp: "Formatted at 2026-09-26 07:34:39 on dist-test-slave-wdjc"
server_key: "9b2175a84f6bb2cf91a96067e104974c"
server_key_iv: "7abc95b7196aa38501a35f5b5944193a"
server_key_version: "encryptionkey@0"
I20260926 07:34:39.712826 12532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407783529717-22743-0/minicluster-data/master-0/wal/instance:
uuid: "4f07f342df5d4f609220ec3f3993d24b"
format_stamp: "Formatted at 2026-09-26 07:34:39 on dist-test-slave-wdjc"
server_key: "9b2175a84f6bb2cf91a96067e104974c"
server_key_iv: "7abc95b7196aa38501a35f5b5944193a"
server_key_version: "encryptionkey@0"
I20260926 07:34:39.717201 12532 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 07:34:39.719949 12547 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:39.721561 12532 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:34:39.721695 12532 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407783529717-22743-0/minicluster-data/master-0/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407783529717-22743-0/minicluster-data/master-0/wal
uuid: "4f07f342df5d4f609220ec3f3993d24b"
format_stamp: "Formatted at 2026-09-26 07:34:39 on dist-test-slave-wdjc"
server_key: "9b2175a84f6bb2cf91a96067e104974c"
server_key_iv: "7abc95b7196aa38501a35f5b5944193a"
server_key_version: "encryptionkey@0"
I20260926 07:34:39.721808 12532 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407783529717-22743-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407783529717-22743-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407783529717-22743-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:34:39.737265 12532 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:34:39.740850 12532 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:34:39.741021 12532 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:34:39.748618 12532 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.254:46639
I20260926 07:34:39.748936 12599 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.254:46639 every 8 connection(s)
I20260926 07:34:39.749706 12532 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
I20260926 07:34:39.750607 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 12532
I20260926 07:34:39.750712 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407783529717-22743-0/minicluster-data/master-0/wal/instance
I20260926 07:34:39.751029 22743 external_mini_cluster.cc:1428] Setting key b10b5f82654198e5bb834a4dcb2ebd66
I20260926 07:34:39.753145 12600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:34:39.757629 12600 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.53.254" port: 46639 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.22.53.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.53.253@KRBTEST.COM" created.
Sep 26 07:34:39 dist-test-slave-wdjc krb5kdc[12519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408079, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.254@KRBTEST.COM for kudu/127.22.53.254@KRBTEST.COM
I20260926 07:34:39.775827 12603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:39.760819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46954 (local address 127.22.53.254:46639)
0926 07:34:39.761440 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:39.761458 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:39.761665 (+ 207us) server_negotiation.cc:407] Connection header received
0926 07:34:39.763232 (+ 1567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:39.763236 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:39.763429 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:39.763588 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:39.764451 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:39.765650 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:39.767199 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:39.767833 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:39.768038 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:39.770662 (+ 2624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:39.770669 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:39.770692 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:39.770714 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:39.772427 (+ 1713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:39.772992 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:39.773005 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:39.773007 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:39.773111 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:39.773428 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:39.773431 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:39.773433 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:39.773854 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:39.774008 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:39.774384 (+ 376us) server_negotiation.cc:299] Negotiation successful
0926 07:34:39.774673 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":491,"spinlock_wait_cycles":41728,"thread_start_us":169,"threads_started":1}
I20260926 07:34:39.775827 12604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:39.760819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:46639 (local address 127.0.0.1:46954)
0926 07:34:39.761472 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:39.761531 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:39.762843 (+ 1312us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:39.763631 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:39.763636 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:39.763973 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:39.764301 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:39.764316 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:39.765793 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:39.765797 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:39.767031 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:39.767046 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:39.768034 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:39.768037 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:39.768145 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:39.768903 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:39.768921 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:39.770494 (+ 1573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:34:39.772602 (+ 2108us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:39.772614 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:39.772626 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:39.772875 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:39.773209 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:39.773214 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:39.773217 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:39.773333 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:39.773975 (+ 642us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:34:39.773981 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:34:39.774131 (+ 150us) client_negotiation.cc:769] Sending connection context
0926 07:34:39.774309 (+ 178us) client_negotiation.cc:240] Negotiation successful
0926 07:34:39.774739 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":383,"mutex_wait_us":111,"spinlock_wait_cycles":896,"thread_start_us":169,"threads_started":1}
I20260926 07:34:39.778759 12600 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.53.253" port: 40235 } has no permanent_uuid. Determining permanent_uuid...
I20260926 07:34:39.779595 12604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:39.779299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.253:40235 (local address 127.0.0.1:33928)
0926 07:34:39.779444 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:39.779522 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.253:40235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260926 07:34:39.779824 12549 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.53.253:40235: connect: Connection refused (error 111)
I20260926 07:34:39.780948 12604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:39.780684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.253:40235 (local address 127.0.0.1:33940)
0926 07:34:39.780824 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:39.780884 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.253:40235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260926 07:34:39.781097 12600 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.53.253:40235: Network error: Client connection negotiation failed: client connection to 127.22.53.253:40235: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.53.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.253.keytab.
Entry for principal kudu/127.22.53.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.253.keytab.
I20260926 07:34:39.783896 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407783529717-22743-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407783529717-22743-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407783529717-22743-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407783529717-22743-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.53.253:40235
--webserver_interface=127.22.53.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.253.keytab
--principal=kudu/127.22.53.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:41329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.53.254:46639,127.22.53.253:40235,127.22.53.252:39951
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:34:39.830416 12600 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.53.253" port: 40235 } attempt: 1
I20260926 07:34:39.831568 12604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:39.831114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.253:40235 (local address 127.0.0.1:33954)
0926 07:34:39.831342 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:39.831439 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.253:40235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260926 07:34:39.832448 12604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:39.832187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.253:40235 (local address 127.0.0.1:33960)
0926 07:34:39.832314 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:39.832371 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.253:40235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260926 07:34:39.832670 12600 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.53.253:40235: Network error: Client connection negotiation failed: client connection to 127.22.53.253:40235: connect: Connection refused (error 111)
I20260926 07:34:39.900879 12600 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.53.253" port: 40235 } attempt: 2
I20260926 07:34:39.901785 12604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:39.901423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.253:40235 (local address 127.0.0.1:33962)
0926 07:34:39.901607 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:39.901677 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.253:40235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260926 07:34:39.902513 12604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:39.902292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.253:40235 (local address 127.0.0.1:33964)
0926 07:34:39.902400 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:39.902451 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.253:40235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260926 07:34:39.902642 12600 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.53.253:40235: Network error: Client connection negotiation failed: client connection to 127.22.53.253:40235: connect: Connection refused (error 111)
W20260926 07:34:39.904635 12607 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 07:34:39.904939 12607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:34:39.904964 12607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:34:39.905012 12607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:34:39.909370 12607 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:34:39.909442 12607 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:34:39.909462 12607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:34:39.909479 12607 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:34:39.909495 12607 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:34:39.915341 12607 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:41329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407783529717-22743-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407783529717-22743-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.22.53.254:46639,127.22.53.253:40235,127.22.53.252:39951
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.253.keytab
--principal=kudu/127.22.53.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.53.253:40235
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407783529717-22743-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.22.53.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12607
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407783529717-22743-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:34:39.916488 12607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:34:39.917635 12607 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:34:39.926795 12613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:39.926797 12615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:39.926841 12612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:34:39.926954 12607 server_base.cc:1034] running on GCE node
I20260926 07:34:39.927668 12607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:34:39.929040 12607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:34:39.930284 12607 hybrid_clock.cc:648] HybridClock initialized: now 1790408079930244 us; error 60 us; skew 500 ppm
Sep 26 07:34:39 dist-test-slave-wdjc krb5kdc[12519](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408079, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:34:39.933772 12607 init.cc:377] Logged in from keytab as kudu/127.22.53.253@KRBTEST.COM (short username kudu)
I20260926 07:34:39.934942 12607 webserver.cc:460] Webserver started at http://127.22.53.253:39663/ using document root <none> and password file <none>
I20260926 07:34:39.935598 12607 fs_manager.cc:359] Metadata directory not provided
I20260926 07:34:39.935702 12607 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:34:39.935963 12607 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:34:39.938385 12607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407783529717-22743-0/minicluster-data/master-1/data/instance:
uuid: "4da98cec4eeb435b822701ea535d1040"
format_stamp: "Formatted at 2026-09-26 07:34:39 on dist-test-slave-wdjc"
server_key: "5041b9f5d513cba57238c5d9d7b79f7b"
server_key_iv: "1d619570b2ec8c6f5d18333cd4209ca1"
server_key_version: "encryptionkey@0"
I20260926 07:34:39.939023 12607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407783529717-22743-0/minicluster-data/master-1/wal/instance:
uuid: "4da98cec4eeb435b822701ea535d1040"
format_stamp: "Formatted at 2026-09-26 07:34:39 on dist-test-slave-wdjc"
server_key: "5041b9f5d513cba57238c5d9d7b79f7b"
server_key_iv: "1d619570b2ec8c6f5d18333cd4209ca1"
server_key_version: "encryptionkey@0"
I20260926 07:34:39.943289 12607 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 07:34:39.946420 12622 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:39.948033 12607 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 07:34:39.948230 12607 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407783529717-22743-0/minicluster-data/master-1/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407783529717-22743-0/minicluster-data/master-1/wal
uuid: "4da98cec4eeb435b822701ea535d1040"
format_stamp: "Formatted at 2026-09-26 07:34:39 on dist-test-slave-wdjc"
server_key: "5041b9f5d513cba57238c5d9d7b79f7b"
server_key_iv: "1d619570b2ec8c6f5d18333cd4209ca1"
server_key_version: "encryptionkey@0"
I20260926 07:34:39.948391 12607 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407783529717-22743-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407783529717-22743-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407783529717-22743-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:34:39.963665 12607 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:34:39.967278 12607 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:34:39.967530 12607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:34:39.975530 12607 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.253:40235
I20260926 07:34:39.975859 12674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.253:40235 every 8 connection(s)
I20260926 07:34:39.976752 12607 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407783529717-22743-0/minicluster-data/master-1/data/info.pb
I20260926 07:34:39.979873 12675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:34:39.981443 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 12607
I20260926 07:34:39.981606 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407783529717-22743-0/minicluster-data/master-1/wal/instance
I20260926 07:34:39.981956 22743 external_mini_cluster.cc:1428] Setting key 7a6b93dfff39e18f5812eff3fd9db551
I20260926 07:34:39.985554 12675 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.53.254" port: 46639 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.22.53.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.53.252@KRBTEST.COM" created.
I20260926 07:34:39.993799 12600 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.53.253" port: 40235 } attempt: 3
Sep 26 07:34:39 dist-test-slave-wdjc krb5kdc[12519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408079, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.253@KRBTEST.COM for kudu/127.22.53.254@KRBTEST.COM
Sep 26 07:34:39 dist-test-slave-wdjc krb5kdc[12519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408079, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.254@KRBTEST.COM for kudu/127.22.53.253@KRBTEST.COM
I20260926 07:34:40.004077 12603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:39.987934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46964 (local address 127.22.53.254:46639)
0926 07:34:39.988088 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:39.988093 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:39.989488 (+ 1395us) server_negotiation.cc:407] Connection header received
0926 07:34:39.990584 (+ 1096us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:39.990587 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:39.990655 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:39.990773 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:39.992189 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:39.992898 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:39.994927 (+ 2029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:39.995309 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:39.995406 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:39.998758 (+ 3352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:39.998764 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:39.998770 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:39.998818 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:40.001306 (+ 2488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:40.002081 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:40.002087 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:40.002091 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:40.002183 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:40.002580 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:40.002585 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:40.002588 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:40.002858 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:40.002969 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:40.003731 (+ 762us) server_negotiation.cc:299] Negotiation successful
0926 07:34:40.003876 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 07:34:40.005158 12678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:39.988618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:46639 (local address 127.0.0.1:46964)
0926 07:34:39.989287 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:39.989339 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:39.990337 (+ 998us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:39.990986 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:39.990996 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:39.991375 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:39.991966 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:39.991988 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:39.993099 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:39.993102 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:39.994787 (+ 1685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:39.994803 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:39.995584 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:39.995591 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:39.995845 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:39.996622 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:39.996650 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:39.998588 (+ 1938us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:34:40.001436 (+ 2848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:40.001448 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:40.001465 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:40.001960 (+ 495us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:40.002289 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:40.002293 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:40.002295 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:40.002467 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:40.003005 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:34:40.003015 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:34:40.003403 (+ 388us) client_negotiation.cc:769] Sending connection context
0926 07:34:40.003665 (+ 262us) client_negotiation.cc:240] Negotiation successful
0926 07:34:40.004073 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":175,"threads_started":1}
I20260926 07:34:40.005327 12604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:39.994373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.253:40235 (local address 127.0.0.1:33968)
0926 07:34:39.994502 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:39.994520 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:39.994603 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:39.995396 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:39.995403 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:39.995639 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:39.995783 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:39.995790 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:39.997033 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:39.997038 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:39.997911 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:39.997924 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:39.998685 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:39.998688 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:39.998822 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:39.999340 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:39.999361 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:40.000968 (+ 1607us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:34:40.003493 (+ 2525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:40.003498 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:40.003500 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:40.003764 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:40.004085 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:40.004091 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:40.004096 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:40.004193 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:40.004780 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:34:40.004783 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:34:40.004896 (+ 113us) client_negotiation.cc:769] Sending connection context
0926 07:34:40.005011 (+ 115us) client_negotiation.cc:240] Negotiation successful
0926 07:34:40.005155 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260926 07:34:40.005759 12679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:39.994478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33968 (local address 127.22.53.253:40235)
0926 07:34:39.994845 (+ 367us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:39.994854 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:39.994901 (+ 47us) server_negotiation.cc:407] Connection header received
0926 07:34:39.994946 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:39.994952 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:39.995190 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:39.995379 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:39.995917 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:39.996912 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:39.998063 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:39.998550 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:39.998675 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:40.001161 (+ 2486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:40.001170 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:40.001192 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:40.001225 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:40.003380 (+ 2155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:40.003890 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:40.003896 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:40.003898 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:40.003984 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:40.004313 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:40.004316 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:40.004317 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:40.004641 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:40.004830 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:40.005097 (+ 267us) server_negotiation.cc:299] Negotiation successful
0926 07:34:40.005235 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":140,"threads_started":1}
I20260926 07:34:40.006722 12675 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.53.253" port: 40235 } has no permanent_uuid. Determining permanent_uuid...
I20260926 07:34:40.008064 12600 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.53.252" port: 39951 } has no permanent_uuid. Determining permanent_uuid...
I20260926 07:34:40.008814 12604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:40.008589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.252:39951 (local address 127.0.0.1:60546)
0926 07:34:40.008686 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:40.008742 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.252:39951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260926 07:34:40.009557 12604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:40.009227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.252:39951 (local address 127.0.0.1:60548)
0926 07:34:40.009431 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:40.009475 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.252:39951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142}
W20260926 07:34:40.009759 12600 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.53.252:39951: Network error: Client connection negotiation failed: client connection to 127.22.53.252:39951: connect: Connection refused (error 111)
Sep 26 07:34:40 dist-test-slave-wdjc krb5kdc[12519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408079, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.253@KRBTEST.COM for kudu/127.22.53.253@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.53.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.252.keytab.
Entry for principal kudu/127.22.53.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.252.keytab.
I20260926 07:34:40.014312 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407783529717-22743-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407783529717-22743-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407783529717-22743-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407783529717-22743-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.53.252:39951
--webserver_interface=127.22.53.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.252.keytab
--principal=kudu/127.22.53.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:41329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.53.254:46639,127.22.53.253:40235,127.22.53.252:39951
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:34:40.016652 12679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:40.007540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33982 (local address 127.22.53.253:40235)
0926 07:34:40.007638 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:40.007642 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:40.007775 (+ 133us) server_negotiation.cc:407] Connection header received
0926 07:34:40.007938 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:40.007941 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:40.007991 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:40.008073 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:40.008527 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:40.009206 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:40.010130 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:40.010408 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:40.010528 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:40.012883 (+ 2355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:40.012887 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:40.012889 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:40.012911 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:40.014718 (+ 1807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:40.015150 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:40.015156 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:40.015159 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:40.015251 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:40.015513 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:40.015519 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:40.015523 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:40.015805 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:40.015898 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:40.016233 (+ 335us) server_negotiation.cc:299] Negotiation successful
0926 07:34:40.016406 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":61,"server-negotiator.queue_time_us":34}
I20260926 07:34:40.016655 12678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:40.007401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.253:40235 (local address 127.0.0.1:33982)
0926 07:34:40.007697 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:40.007712 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:40.007830 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:40.008081 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:40.008084 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:40.008280 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:40.008392 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:40.008400 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:40.009316 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:40.009320 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:40.010025 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:40.010034 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:40.010515 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:40.010517 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:40.010662 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:40.011126 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:40.011138 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:40.012720 (+ 1582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:34:40.014836 (+ 2116us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:40.014838 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:40.014840 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:40.015049 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:40.015346 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:40.015348 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:40.015349 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:40.015425 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:40.015924 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:34:40.015929 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:34:40.016047 (+ 118us) client_negotiation.cc:769] Sending connection context
0926 07:34:40.016168 (+ 121us) client_negotiation.cc:240] Negotiation successful
0926 07:34:40.016348 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175}
I20260926 07:34:40.017876 12675 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.53.252" port: 39951 } has no permanent_uuid. Determining permanent_uuid...
I20260926 07:34:40.018796 12678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:40.018411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.252:39951 (local address 127.0.0.1:60564)
0926 07:34:40.018578 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:40.018691 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.252:39951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260926 07:34:40.019099 12623 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.53.252:39951: connect: Connection refused (error 111)
I20260926 07:34:40.020136 12678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:40.019836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.252:39951 (local address 127.0.0.1:60576)
0926 07:34:40.019980 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:40.020050 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.252:39951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260926 07:34:40.020371 12675 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.53.252:39951: Network error: Client connection negotiation failed: client connection to 127.22.53.252:39951: connect: Connection refused (error 111)
I20260926 07:34:40.040932 12600 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.53.252" port: 39951 } attempt: 1
I20260926 07:34:40.041878 12604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:40.041524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.252:39951 (local address 127.0.0.1:60590)
0926 07:34:40.041700 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:40.041774 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.252:39951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260926 07:34:40.042725 12604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:40.042488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.252:39951 (local address 127.0.0.1:60602)
0926 07:34:40.042604 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:40.042658 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.252:39951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260926 07:34:40.042922 12600 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.53.252:39951: Network error: Client connection negotiation failed: client connection to 127.22.53.252:39951: connect: Connection refused (error 111)
I20260926 07:34:40.069705 12675 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.53.252" port: 39951 } attempt: 1
I20260926 07:34:40.070648 12678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:40.070319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.252:39951 (local address 127.0.0.1:60610)
0926 07:34:40.070502 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:40.070566 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.252:39951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260926 07:34:40.071439 12678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:40.071223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.252:39951 (local address 127.0.0.1:60626)
0926 07:34:40.071329 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:40.071385 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.252:39951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260926 07:34:40.071569 12675 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.53.252:39951: Network error: Client connection negotiation failed: client connection to 127.22.53.252:39951: connect: Connection refused (error 111)
I20260926 07:34:40.118149 12600 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.53.252" port: 39951 } attempt: 2
I20260926 07:34:40.119158 12604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:40.118827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.252:39951 (local address 127.0.0.1:60640)
0926 07:34:40.119005 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:40.119071 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.252:39951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260926 07:34:40.120041 12604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:40.119782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.252:39951 (local address 127.0.0.1:60650)
0926 07:34:40.119919 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:40.119977 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.252:39951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260926 07:34:40.120190 12600 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.53.252:39951: Network error: Client connection negotiation failed: client connection to 127.22.53.252:39951: connect: Connection refused (error 111)
W20260926 07:34:40.133522 12681 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 07:34:40.133911 12681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:34:40.133980 12681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:34:40.134084 12681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:34:40.138496 12681 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:34:40.138612 12681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:34:40.138646 12681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:34:40.138687 12681 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:34:40.138721 12681 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:34:40.139739 12675 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.53.252" port: 39951 } attempt: 2
I20260926 07:34:40.140686 12678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:40.140312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.252:39951 (local address 127.0.0.1:60662)
0926 07:34:40.140468 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:40.140597 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.252:39951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260926 07:34:40.141453 12678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:40.141225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.252:39951 (local address 127.0.0.1:60666)
0926 07:34:40.141331 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:40.141384 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.252:39951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260926 07:34:40.141582 12675 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.53.252:39951: Network error: Client connection negotiation failed: client connection to 127.22.53.252:39951: connect: Connection refused (error 111)
I20260926 07:34:40.145020 12681 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:41329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407783529717-22743-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407783529717-22743-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.22.53.254:46639,127.22.53.253:40235,127.22.53.252:39951
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.252.keytab
--principal=kudu/127.22.53.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.53.252:39951
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407783529717-22743-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.22.53.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12681
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407783529717-22743-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:34:40.146246 12681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:34:40.147346 12681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:34:40.156428 12690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:34:40.156550 12681 server_base.cc:1034] running on GCE node
W20260926 07:34:40.156428 12688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:40.156462 12687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:34:40.157274 12681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:34:40.158519 12681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:34:40.159721 12681 hybrid_clock.cc:648] HybridClock initialized: now 1790408080159711 us; error 30 us; skew 500 ppm
Sep 26 07:34:40 dist-test-slave-wdjc krb5kdc[12519](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408080, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:34:40.162776 12681 init.cc:377] Logged in from keytab as kudu/127.22.53.252@KRBTEST.COM (short username kudu)
I20260926 07:34:40.163798 12681 webserver.cc:460] Webserver started at http://127.22.53.252:39501/ using document root <none> and password file <none>
I20260926 07:34:40.164351 12681 fs_manager.cc:359] Metadata directory not provided
I20260926 07:34:40.164420 12681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:34:40.164664 12681 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:34:40.166805 12681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407783529717-22743-0/minicluster-data/master-2/data/instance:
uuid: "1bc11ca2eadd4b75b7ed372055a0fd56"
format_stamp: "Formatted at 2026-09-26 07:34:40 on dist-test-slave-wdjc"
server_key: "d80dae444d989219370011a3a4ddd98b"
server_key_iv: "35b6a77b83c68ecb751e16b791ff5afe"
server_key_version: "encryptionkey@0"
I20260926 07:34:40.167392 12681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407783529717-22743-0/minicluster-data/master-2/wal/instance:
uuid: "1bc11ca2eadd4b75b7ed372055a0fd56"
format_stamp: "Formatted at 2026-09-26 07:34:40 on dist-test-slave-wdjc"
server_key: "d80dae444d989219370011a3a4ddd98b"
server_key_iv: "35b6a77b83c68ecb751e16b791ff5afe"
server_key_version: "encryptionkey@0"
I20260926 07:34:40.171649 12681 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260926 07:34:40.174486 12697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:40.176060 12681 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 07:34:40.176230 12681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407783529717-22743-0/minicluster-data/master-2/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407783529717-22743-0/minicluster-data/master-2/wal
uuid: "1bc11ca2eadd4b75b7ed372055a0fd56"
format_stamp: "Formatted at 2026-09-26 07:34:40 on dist-test-slave-wdjc"
server_key: "d80dae444d989219370011a3a4ddd98b"
server_key_iv: "35b6a77b83c68ecb751e16b791ff5afe"
server_key_version: "encryptionkey@0"
I20260926 07:34:40.176390 12681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407783529717-22743-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407783529717-22743-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407783529717-22743-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:34:40.198127 12681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:34:40.201562 12681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:34:40.201736 12681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:34:40.209021 12681 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.252:39951
I20260926 07:34:40.209322 12749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.252:39951 every 8 connection(s)
I20260926 07:34:40.210120 12681 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407783529717-22743-0/minicluster-data/master-2/data/info.pb
I20260926 07:34:40.212344 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 12681
I20260926 07:34:40.212453 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407783529717-22743-0/minicluster-data/master-2/wal/instance
I20260926 07:34:40.212805 22743 external_mini_cluster.cc:1428] Setting key f227846e67b2b8331d2a3b898ef7f3a1
I20260926 07:34:40.213582 12750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:34:40.218739 12750 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.53.254" port: 46639 } has no permanent_uuid. Determining permanent_uuid...
I20260926 07:34:40.219333 12600 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.53.252" port: 39951 } attempt: 3
Sep 26 07:34:40 dist-test-slave-wdjc krb5kdc[12519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408079, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.254@KRBTEST.COM for kudu/127.22.53.252@KRBTEST.COM
Sep 26 07:34:40 dist-test-slave-wdjc krb5kdc[12519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408079, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.53.252@KRBTEST.COM
Sep 26 07:34:40 dist-test-slave-wdjc krb5kdc[12519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408080, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.252@KRBTEST.COM for kudu/127.22.53.254@KRBTEST.COM
I20260926 07:34:40.230970 12604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:40.219875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.252:39951 (local address 127.0.0.1:60686)
0926 07:34:40.220015 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:40.220032 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:40.220113 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:40.220731 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:40.220735 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:40.220964 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:40.221074 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:40.221083 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:40.222199 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:40.222205 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:40.222923 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:40.222934 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:40.223614 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:40.223617 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:40.223702 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:40.224282 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:40.224297 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:40.226061 (+ 1764us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:34:40.228473 (+ 2412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:40.228477 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:40.228479 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:40.228774 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:40.229205 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:40.229210 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:40.229215 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:40.229319 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:40.230268 (+ 949us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:34:40.230272 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:34:40.230505 (+ 233us) client_negotiation.cc:769] Sending connection context
0926 07:34:40.230612 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 07:34:40.230778 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260926 07:34:40.232719 12675 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.53.252" port: 39951 } attempt: 3
I20260926 07:34:40.233201 12754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:40.219939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60686 (local address 127.22.53.252:39951)
0926 07:34:40.220325 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:40.220335 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:40.220349 (+ 14us) server_negotiation.cc:407] Connection header received
0926 07:34:40.220384 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:40.220386 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:40.220425 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:40.220605 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:40.221244 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:40.222070 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:40.223056 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:40.223477 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:40.223947 (+ 470us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:40.226279 (+ 2332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:40.226285 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:40.226326 (+ 41us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:40.226372 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:40.228312 (+ 1940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:40.228909 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:40.228916 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:40.228923 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:40.229078 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:40.229452 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:40.229457 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:40.229459 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:40.230130 (+ 671us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:40.231400 (+ 1270us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:40.231612 (+ 212us) server_negotiation.cc:299] Negotiation successful
0926 07:34:40.231980 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":321,"thread_start_us":181,"threads_started":1}
I20260926 07:34:40.235502 12753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:40.214318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60670 (local address 127.22.53.252:39951)
0926 07:34:40.214803 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:40.214830 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:40.214876 (+ 46us) server_negotiation.cc:407] Connection header received
0926 07:34:40.215803 (+ 927us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:40.215829 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:40.216320 (+ 491us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:40.216772 (+ 452us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:40.217874 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:40.219706 (+ 1832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:40.222676 (+ 2970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:40.223440 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:40.223612 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:40.228971 (+ 5359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:40.228975 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:40.228978 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:40.229007 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:40.232381 (+ 3374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:40.233482 (+ 1101us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:40.233487 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:40.233489 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:40.233598 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:40.233997 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:40.234000 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:40.234001 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:40.234257 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:40.234424 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:40.235118 (+ 694us) server_negotiation.cc:299] Negotiation successful
0926 07:34:40.235263 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":325,"thread_start_us":139,"threads_started":1}
I20260926 07:34:40.235771 12603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:40.221135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46966 (local address 127.22.53.254:46639)
0926 07:34:40.221266 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:40.221270 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:40.221701 (+ 431us) server_negotiation.cc:407] Connection header received
0926 07:34:40.221946 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:40.221951 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:40.222013 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:40.222074 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:40.223098 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:40.223760 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:40.225119 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:40.225470 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:40.225546 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:40.228921 (+ 3375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:40.228925 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:40.228927 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:40.228951 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:40.232278 (+ 3327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:40.232962 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:40.232966 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:40.232968 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:40.233060 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:40.234587 (+ 1527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:40.234590 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:40.234592 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:40.234930 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:40.235021 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:40.235442 (+ 421us) server_negotiation.cc:299] Negotiation successful
0926 07:34:40.235588 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 07:34:40.236672 12755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:40.221060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:46639 (local address 127.0.0.1:46966)
0926 07:34:40.221555 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:40.221623 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:40.221798 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:40.222310 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:40.222319 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:40.222823 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:40.222959 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:40.222966 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:40.223902 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:40.223908 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:40.224967 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:40.224980 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:40.225804 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:40.225809 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:40.225941 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:40.226472 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:40.226486 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:40.228741 (+ 2255us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:34:40.232413 (+ 3672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:40.232421 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:40.232427 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:40.232764 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:40.233176 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:40.233179 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:40.233181 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:40.233277 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:40.235088 (+ 1811us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:34:40.235093 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:34:40.235210 (+ 117us) client_negotiation.cc:769] Sending connection context
0926 07:34:40.236284 (+ 1074us) client_negotiation.cc:240] Negotiation successful
0926 07:34:40.236463 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":112,"threads_started":1}
I20260926 07:34:40.237900 12750 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.53.253" port: 40235 } has no permanent_uuid. Determining permanent_uuid...
I20260926 07:34:40.239043 12600 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f07f342df5d4f609220ec3f3993d24b: Bootstrap starting.
Sep 26 07:34:40 dist-test-slave-wdjc krb5kdc[12519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408079, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.253@KRBTEST.COM for kudu/127.22.53.252@KRBTEST.COM
I20260926 07:34:40.242782 12600 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4f07f342df5d4f609220ec3f3993d24b: Neither blocks nor log segments found. Creating new log.
I20260926 07:34:40.243872 12600 log.cc:824] T 00000000000000000000000000000000 P 4f07f342df5d4f609220ec3f3993d24b: Log is configured to *not* fsync() on all Append() calls
Sep 26 07:34:40 dist-test-slave-wdjc krb5kdc[12519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408080, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.252@KRBTEST.COM for kudu/127.22.53.253@KRBTEST.COM
I20260926 07:34:40.246112 12678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:40.235537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.252:39951 (local address 127.0.0.1:60694)
0926 07:34:40.235766 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:40.235787 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:40.235888 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:40.236965 (+ 1077us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:40.236969 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:40.237172 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:40.237346 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:40.237356 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:40.238171 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:40.238175 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:40.238840 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:40.238851 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:40.239439 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:40.239443 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:40.239533 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:40.240089 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:40.240101 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:40.241758 (+ 1657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:34:40.244193 (+ 2435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:40.244196 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:40.244199 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:40.244464 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:40.244922 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:40.244928 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:40.244933 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:40.245043 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:40.245541 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:34:40.245544 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:34:40.245663 (+ 119us) client_negotiation.cc:769] Sending connection context
0926 07:34:40.245760 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 07:34:40.245923 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132}
I20260926 07:34:40.246863 12753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:40.236179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60694 (local address 127.22.53.252:39951)
0926 07:34:40.236698 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:40.236703 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:40.236719 (+ 16us) server_negotiation.cc:407] Connection header received
0926 07:34:40.236755 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:40.236758 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:40.236807 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:40.237466 (+ 659us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:40.237540 (+ 74us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:40.238049 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:40.238970 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:40.239315 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:40.239425 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:40.241954 (+ 2529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:40.241959 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:40.241962 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:40.241989 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:40.243951 (+ 1962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:40.244663 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:40.244671 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:40.244675 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:40.244780 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:40.245186 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:40.245190 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:40.245192 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:40.245417 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:40.246468 (+ 1051us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:40.246550 (+ 82us) server_negotiation.cc:299] Negotiation successful
0926 07:34:40.246700 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439}
I20260926 07:34:40.247085 12600 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f07f342df5d4f609220ec3f3993d24b: No bootstrap required, opened a new log
Sep 26 07:34:40 dist-test-slave-wdjc krb5kdc[12519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408079, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.53.254@KRBTEST.COM
I20260926 07:34:40.250317 12675 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4da98cec4eeb435b822701ea535d1040: Bootstrap starting.
I20260926 07:34:40.250667 12679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:40.238534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33994 (local address 127.22.53.253:40235)
0926 07:34:40.240711 (+ 2177us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:40.240718 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:40.240739 (+ 21us) server_negotiation.cc:407] Connection header received
0926 07:34:40.240788 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:40.240793 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:40.240842 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:40.240944 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:40.241473 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:40.242185 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:40.243282 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:40.243634 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:40.243702 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:40.246426 (+ 2724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:40.246431 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:40.246435 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:40.246462 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:40.248470 (+ 2008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:40.249047 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:40.249051 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:40.249053 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:40.249142 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:40.249504 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:40.249509 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:40.249511 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:40.249811 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:40.249900 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:40.250341 (+ 441us) server_negotiation.cc:299] Negotiation successful
0926 07:34:40.250471 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2071}
I20260926 07:34:40.251647 12755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:40.238417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.253:40235 (local address 127.0.0.1:33994)
0926 07:34:40.239049 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:40.239062 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:40.239122 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:40.240984 (+ 1862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:40.240988 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:40.241184 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:40.241327 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:40.241335 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:40.242329 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:40.242333 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:40.243140 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:40.243151 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:40.243868 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:40.243871 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:40.243962 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:40.244669 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:40.244684 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:40.246261 (+ 1577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:34:40.248685 (+ 2424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:40.248688 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:40.248691 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:40.248934 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:40.249279 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:40.249285 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:40.249290 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:40.249385 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:40.249980 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:34:40.249986 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:34:40.250110 (+ 124us) client_negotiation.cc:769] Sending connection context
0926 07:34:40.251269 (+ 1159us) client_negotiation.cc:240] Negotiation successful
0926 07:34:40.251445 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"spinlock_wait_cycles":64896}
I20260926 07:34:40.251610 12600 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4f07f342df5d4f609220ec3f3993d24b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f07f342df5d4f609220ec3f3993d24b" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 46639 } } peers { permanent_uuid: "4da98cec4eeb435b822701ea535d1040" member_type: VOTER last_known_addr { host: "127.22.53.253" port: 40235 } } peers { permanent_uuid: "1bc11ca2eadd4b75b7ed372055a0fd56" member_type: VOTER last_known_addr { host: "127.22.53.252" port: 39951 } }
I20260926 07:34:40.251899 12600 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4f07f342df5d4f609220ec3f3993d24b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f07f342df5d4f609220ec3f3993d24b, State: Initialized, Role: FOLLOWER
I20260926 07:34:40.252336 12600 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4f07f342df5d4f609220ec3f3993d24b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f07f342df5d4f609220ec3f3993d24b" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 46639 } } peers { permanent_uuid: "4da98cec4eeb435b822701ea535d1040" member_type: VOTER last_known_addr { host: "127.22.53.253" port: 40235 } } peers { permanent_uuid: "1bc11ca2eadd4b75b7ed372055a0fd56" member_type: VOTER last_known_addr { host: "127.22.53.252" port: 39951 } }
I20260926 07:34:40.252838 12750 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.53.252" port: 39951 } has no permanent_uuid. Determining permanent_uuid...
I20260926 07:34:40.253052 12603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:40.241498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46980 (local address 127.22.53.254:46639)
0926 07:34:40.241612 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:40.241616 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:40.241799 (+ 183us) server_negotiation.cc:407] Connection header received
0926 07:34:40.241984 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:40.241987 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:40.242032 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:40.242078 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:40.242989 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:40.243728 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:40.244891 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:40.245272 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:40.245357 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:40.249048 (+ 3691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:40.249052 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:40.249054 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:40.249083 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:40.250935 (+ 1852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:40.251496 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:40.251500 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:40.251502 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:40.251588 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:40.251944 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:40.251948 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:40.251950 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:40.252180 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:40.252335 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:40.252758 (+ 423us) server_negotiation.cc:299] Negotiation successful
0926 07:34:40.252897 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 07:34:40.253666 12675 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4da98cec4eeb435b822701ea535d1040: Neither blocks nor log segments found. Creating new log.
I20260926 07:34:40.254027 12757 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f07f342df5d4f609220ec3f3993d24b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f07f342df5d4f609220ec3f3993d24b" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 46639 } } peers { permanent_uuid: "4da98cec4eeb435b822701ea535d1040" member_type: VOTER last_known_addr { host: "127.22.53.253" port: 40235 } } peers { permanent_uuid: "1bc11ca2eadd4b75b7ed372055a0fd56" member_type: VOTER last_known_addr { host: "127.22.53.252" port: 39951 } } }
I20260926 07:34:40.254277 12757 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f07f342df5d4f609220ec3f3993d24b [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:34:40.254650 12675 log.cc:824] T 00000000000000000000000000000000 P 4da98cec4eeb435b822701ea535d1040: Log is configured to *not* fsync() on all Append() calls
I20260926 07:34:40.255054 12600 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4f07f342df5d4f609220ec3f3993d24b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:34:40.257820 12675 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4da98cec4eeb435b822701ea535d1040: No bootstrap required, opened a new log
Sep 26 07:34:40 dist-test-slave-wdjc krb5kdc[12519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408079, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.53.253@KRBTEST.COM
I20260926 07:34:40.262257 12675 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4da98cec4eeb435b822701ea535d1040 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f07f342df5d4f609220ec3f3993d24b" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 46639 } } peers { permanent_uuid: "4da98cec4eeb435b822701ea535d1040" member_type: VOTER last_known_addr { host: "127.22.53.253" port: 40235 } } peers { permanent_uuid: "1bc11ca2eadd4b75b7ed372055a0fd56" member_type: VOTER last_known_addr { host: "127.22.53.252" port: 39951 } }
I20260926 07:34:40.262537 12675 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4da98cec4eeb435b822701ea535d1040 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4da98cec4eeb435b822701ea535d1040, State: Initialized, Role: FOLLOWER
Sep 26 07:34:40 dist-test-slave-wdjc krb5kdc[12519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408080, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.252@KRBTEST.COM for kudu/127.22.53.252@KRBTEST.COM
I20260926 07:34:40.262912 12675 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4da98cec4eeb435b822701ea535d1040 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f07f342df5d4f609220ec3f3993d24b" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 46639 } } peers { permanent_uuid: "4da98cec4eeb435b822701ea535d1040" member_type: VOTER last_known_addr { host: "127.22.53.253" port: 40235 } } peers { permanent_uuid: "1bc11ca2eadd4b75b7ed372055a0fd56" member_type: VOTER last_known_addr { host: "127.22.53.252" port: 39951 } }
I20260926 07:34:40.260377 12532 master_runner.cc:186] Error getting master registration for 127.22.53.253:40235: OK, instance_id { permanent_uuid: "4da98cec4eeb435b822701ea535d1040" instance_seqno: 1790408079973274 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 07:34:40.264441 12675 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4da98cec4eeb435b822701ea535d1040 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:34:40.264485 12532 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 07:34:40.266912 12755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:40.253405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.252:39951 (local address 127.0.0.1:60704)
0926 07:34:40.253547 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:40.253563 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:40.253644 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:40.257727 (+ 4083us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:40.257730 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:40.257937 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:40.258055 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:40.258066 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:40.259119 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:40.259123 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:40.260045 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:40.260062 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:40.260738 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:40.260742 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:40.260946 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:40.261546 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:40.261561 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:40.263174 (+ 1613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:34:40.265199 (+ 2025us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:40.265204 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:40.265206 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:40.265449 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:40.265747 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:40.265750 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:40.265752 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:40.265827 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:40.266235 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:34:40.266238 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:34:40.266424 (+ 186us) client_negotiation.cc:769] Sending connection context
0926 07:34:40.266563 (+ 139us) client_negotiation.cc:240] Negotiation successful
0926 07:34:40.266711 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260926 07:34:40.267150 12679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:40.254648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34004 (local address 127.22.53.253:40235)
0926 07:34:40.254755 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:40.254759 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:40.254842 (+ 83us) server_negotiation.cc:407] Connection header received
0926 07:34:40.254960 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:40.254963 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:40.255008 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:40.255111 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:40.255742 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:40.256555 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:40.257677 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:40.258012 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:40.258131 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:40.262162 (+ 4031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:40.262167 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:40.262170 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:40.262201 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:40.264580 (+ 2379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:40.265142 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:40.265148 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:40.265151 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:40.265262 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:40.265580 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:40.265583 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:40.265585 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:40.265829 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:40.266033 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:40.266850 (+ 817us) server_negotiation.cc:299] Negotiation successful
0926 07:34:40.266975 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":42}
I20260926 07:34:40.267629 12753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:40.253587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60704 (local address 127.22.53.252:39951)
0926 07:34:40.257459 (+ 3872us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:40.257465 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:40.257483 (+ 18us) server_negotiation.cc:407] Connection header received
0926 07:34:40.257527 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:40.257531 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:40.257593 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:40.258179 (+ 586us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:40.258269 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:40.258985 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:40.260232 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:40.260588 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:40.261208 (+ 620us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:40.263330 (+ 2122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:40.263335 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:40.263339 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:40.263366 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:40.265098 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:40.265561 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:40.265566 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:40.265568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:40.265642 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:40.265924 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:40.265927 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:40.265928 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:40.266135 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:40.267199 (+ 1064us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:40.267358 (+ 159us) server_negotiation.cc:299] Negotiation successful
0926 07:34:40.267490 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3776}
W20260926 07:34:40.269376 12769 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4f07f342df5d4f609220ec3f3993d24b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:34:40.269487 12769 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:34:40.271071 12750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bc11ca2eadd4b75b7ed372055a0fd56: Bootstrap starting.
I20260926 07:34:40.271410 22743 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:34:40.272712 12770 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4da98cec4eeb435b822701ea535d1040 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f07f342df5d4f609220ec3f3993d24b" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 46639 } } peers { permanent_uuid: "4da98cec4eeb435b822701ea535d1040" member_type: VOTER last_known_addr { host: "127.22.53.253" port: 40235 } } peers { permanent_uuid: "1bc11ca2eadd4b75b7ed372055a0fd56" member_type: VOTER last_known_addr { host: "127.22.53.252" port: 39951 } } }
I20260926 07:34:40.272962 12770 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4da98cec4eeb435b822701ea535d1040 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:34:40.272985 12574 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "4f07f342df5d4f609220ec3f3993d24b"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46980
I20260926 07:34:40.273217 12574 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4f07f342df5d4f609220ec3f3993d24b [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 07:34:40.273343 12574 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4f07f342df5d4f609220ec3f3993d24b [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:34:40.273660 12750 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1bc11ca2eadd4b75b7ed372055a0fd56: Neither blocks nor log segments found. Creating new log.
I20260926 07:34:40.274628 12750 log.cc:824] T 00000000000000000000000000000000 P 1bc11ca2eadd4b75b7ed372055a0fd56: Log is configured to *not* fsync() on all Append() calls
I20260926 07:34:40.274722 12574 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4f07f342df5d4f609220ec3f3993d24b [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f07f342df5d4f609220ec3f3993d24b" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 46639 } } peers { permanent_uuid: "4da98cec4eeb435b822701ea535d1040" member_type: VOTER last_known_addr { host: "127.22.53.253" port: 40235 } } peers { permanent_uuid: "1bc11ca2eadd4b75b7ed372055a0fd56" member_type: VOTER last_known_addr { host: "127.22.53.252" port: 39951 } }
I20260926 07:34:40.276197 12649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4f07f342df5d4f609220ec3f3993d24b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "4da98cec4eeb435b822701ea535d1040"
I20260926 07:34:40.276450 12649 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4da98cec4eeb435b822701ea535d1040 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:34:40.276923 12750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bc11ca2eadd4b75b7ed372055a0fd56: No bootstrap required, opened a new log
I20260926 07:34:40.278390 12649 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4da98cec4eeb435b822701ea535d1040 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f07f342df5d4f609220ec3f3993d24b in term 1.
I20260926 07:34:40.278882 12549 leader_election.cc:302] T 00000000000000000000000000000000 P 4f07f342df5d4f609220ec3f3993d24b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4da98cec4eeb435b822701ea535d1040, 4f07f342df5d4f609220ec3f3993d24b; no voters:
I20260926 07:34:40.279219 12757 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4f07f342df5d4f609220ec3f3993d24b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:34:40.279475 12757 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4f07f342df5d4f609220ec3f3993d24b [term 1 LEADER]: Becoming Leader. State: Replica: 4f07f342df5d4f609220ec3f3993d24b, State: Running, Role: LEADER
I20260926 07:34:40.276475 12607 master_runner.cc:186] Error getting master registration for 127.22.53.252:39951: OK, instance_id { permanent_uuid: "1bc11ca2eadd4b75b7ed372055a0fd56" instance_seqno: 1790408080206894 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 07:34:40.279889 12607 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 07:34:40.279847 12757 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4f07f342df5d4f609220ec3f3993d24b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f07f342df5d4f609220ec3f3993d24b" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 46639 } } peers { permanent_uuid: "4da98cec4eeb435b822701ea535d1040" member_type: VOTER last_known_addr { host: "127.22.53.253" port: 40235 } } peers { permanent_uuid: "1bc11ca2eadd4b75b7ed372055a0fd56" member_type: VOTER last_known_addr { host: "127.22.53.252" port: 39951 } }
I20260926 07:34:40.279979 12574 leader_election.cc:288] T 00000000000000000000000000000000 P 4f07f342df5d4f609220ec3f3993d24b [CANDIDATE]: Term 1 election: Requested vote from peers 4da98cec4eeb435b822701ea535d1040 (127.22.53.253:40235), 1bc11ca2eadd4b75b7ed372055a0fd56 (127.22.53.252:39951)
I20260926 07:34:40.281181 12750 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1bc11ca2eadd4b75b7ed372055a0fd56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f07f342df5d4f609220ec3f3993d24b" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 46639 } } peers { permanent_uuid: "4da98cec4eeb435b822701ea535d1040" member_type: VOTER last_known_addr { host: "127.22.53.253" port: 40235 } } peers { permanent_uuid: "1bc11ca2eadd4b75b7ed372055a0fd56" member_type: VOTER last_known_addr { host: "127.22.53.252" port: 39951 } }
I20260926 07:34:40.281448 12750 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1bc11ca2eadd4b75b7ed372055a0fd56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bc11ca2eadd4b75b7ed372055a0fd56, State: Initialized, Role: FOLLOWER
I20260926 07:34:40.281785 12750 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1bc11ca2eadd4b75b7ed372055a0fd56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f07f342df5d4f609220ec3f3993d24b" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 46639 } } peers { permanent_uuid: "4da98cec4eeb435b822701ea535d1040" member_type: VOTER last_known_addr { host: "127.22.53.253" port: 40235 } } peers { permanent_uuid: "1bc11ca2eadd4b75b7ed372055a0fd56" member_type: VOTER last_known_addr { host: "127.22.53.252" port: 39951 } }
I20260926 07:34:40.282378 12790 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bc11ca2eadd4b75b7ed372055a0fd56 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f07f342df5d4f609220ec3f3993d24b" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 46639 } } peers { permanent_uuid: "4da98cec4eeb435b822701ea535d1040" member_type: VOTER last_known_addr { host: "127.22.53.253" port: 40235 } } peers { permanent_uuid: "1bc11ca2eadd4b75b7ed372055a0fd56" member_type: VOTER last_known_addr { host: "127.22.53.252" port: 39951 } } }
I20260926 07:34:40.282573 12790 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bc11ca2eadd4b75b7ed372055a0fd56 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:34:40.282934 12785 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f07f342df5d4f609220ec3f3993d24b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f07f342df5d4f609220ec3f3993d24b. Latest consensus state: current_term: 1 leader_uuid: "4f07f342df5d4f609220ec3f3993d24b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f07f342df5d4f609220ec3f3993d24b" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 46639 } } peers { permanent_uuid: "4da98cec4eeb435b822701ea535d1040" member_type: VOTER last_known_addr { host: "127.22.53.253" port: 40235 } } peers { permanent_uuid: "1bc11ca2eadd4b75b7ed372055a0fd56" member_type: VOTER last_known_addr { host: "127.22.53.252" port: 39951 } } }
I20260926 07:34:40.283141 12785 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f07f342df5d4f609220ec3f3993d24b [sys.catalog]: This master's current role is: LEADER
I20260926 07:34:40.283298 12750 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1bc11ca2eadd4b75b7ed372055a0fd56 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:34:40.283322 12722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4f07f342df5d4f609220ec3f3993d24b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "1bc11ca2eadd4b75b7ed372055a0fd56"
I20260926 07:34:40.283537 12722 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1bc11ca2eadd4b75b7ed372055a0fd56 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:34:40.285167 12722 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1bc11ca2eadd4b75b7ed372055a0fd56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f07f342df5d4f609220ec3f3993d24b in term 1.
W20260926 07:34:40.289347 12782 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4da98cec4eeb435b822701ea535d1040: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:34:40.289783 12782 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:34:40.291911 12793 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:34:40.293861 12793 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:34:40.295817 12603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:40.282579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46988 (local address 127.22.53.254:46639)
0926 07:34:40.282739 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:40.282747 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:40.285903 (+ 3156us) server_negotiation.cc:407] Connection header received
0926 07:34:40.286107 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:40.286110 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:40.286161 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:40.286211 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:40.287226 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:40.287949 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:40.289100 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:40.289407 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:40.289485 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:40.291335 (+ 1850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:40.291339 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:40.291342 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:40.291372 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:40.293307 (+ 1935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:40.293793 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:40.293797 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:40.293799 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:40.293892 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:40.294198 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:40.294205 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:40.294206 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:40.294463 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:40.294608 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:40.295397 (+ 789us) server_negotiation.cc:299] Negotiation successful
0926 07:34:40.295536 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 07:34:40.297839 12679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:40.283397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34008 (local address 127.22.53.253:40235)
0926 07:34:40.283525 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:40.283529 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:40.283911 (+ 382us) server_negotiation.cc:407] Connection header received
0926 07:34:40.284111 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:40.284114 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:40.284166 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:40.284222 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:40.285286 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:40.286085 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:40.287326 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:40.287690 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:40.287773 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:40.289760 (+ 1987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:40.289763 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:40.289768 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:40.289801 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:40.291957 (+ 2156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:40.295520 (+ 3563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:40.295524 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:40.295526 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:40.295666 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:40.296318 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:40.296321 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:40.296323 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:40.296781 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:40.296959 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:40.297580 (+ 621us) server_negotiation.cc:299] Negotiation successful
0926 07:34:40.297704 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260926 07:34:40.303076 12805 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1bc11ca2eadd4b75b7ed372055a0fd56: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260926 07:34:40.307590 12753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:40.288319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60720 (local address 127.22.53.252:39951)
0926 07:34:40.288447 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:40.288452 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:40.296003 (+ 7551us) server_negotiation.cc:407] Connection header received
0926 07:34:40.296195 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:40.296198 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:40.296246 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:40.296318 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:40.297489 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:40.298267 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:40.300329 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:40.300659 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:40.300790 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:40.302906 (+ 2116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:40.302911 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:40.302914 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:40.302948 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:40.304970 (+ 2022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:40.305614 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:40.305619 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:40.305623 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:40.305745 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:40.306069 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:40.306073 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:40.306075 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:40.306330 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:40.306463 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:40.307249 (+ 786us) server_negotiation.cc:299] Negotiation successful
0926 07:34:40.307398 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260926 07:34:40.308763 12805 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:34:40.313156 12722 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1bc11ca2eadd4b75b7ed372055a0fd56 [term 1 FOLLOWER]: Refusing update from remote peer 4f07f342df5d4f609220ec3f3993d24b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:34:40.313226 12649 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4da98cec4eeb435b822701ea535d1040 [term 1 FOLLOWER]: Refusing update from remote peer 4f07f342df5d4f609220ec3f3993d24b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:34:40.314093 12757 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4f07f342df5d4f609220ec3f3993d24b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da98cec4eeb435b822701ea535d1040" member_type: VOTER last_known_addr { host: "127.22.53.253" port: 40235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:34:40.314647 12757 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4f07f342df5d4f609220ec3f3993d24b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bc11ca2eadd4b75b7ed372055a0fd56" member_type: VOTER last_known_addr { host: "127.22.53.252" port: 39951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:34:40.319607 12790 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bc11ca2eadd4b75b7ed372055a0fd56 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f07f342df5d4f609220ec3f3993d24b. Latest consensus state: current_term: 1 leader_uuid: "4f07f342df5d4f609220ec3f3993d24b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f07f342df5d4f609220ec3f3993d24b" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 46639 } } peers { permanent_uuid: "4da98cec4eeb435b822701ea535d1040" member_type: VOTER last_known_addr { host: "127.22.53.253" port: 40235 } } peers { permanent_uuid: "1bc11ca2eadd4b75b7ed372055a0fd56" member_type: VOTER last_known_addr { host: "127.22.53.252" port: 39951 } } }
I20260926 07:34:40.319607 12770 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4da98cec4eeb435b822701ea535d1040 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f07f342df5d4f609220ec3f3993d24b. Latest consensus state: current_term: 1 leader_uuid: "4f07f342df5d4f609220ec3f3993d24b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f07f342df5d4f609220ec3f3993d24b" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 46639 } } peers { permanent_uuid: "4da98cec4eeb435b822701ea535d1040" member_type: VOTER last_known_addr { host: "127.22.53.253" port: 40235 } } peers { permanent_uuid: "1bc11ca2eadd4b75b7ed372055a0fd56" member_type: VOTER last_known_addr { host: "127.22.53.252" port: 39951 } } }
I20260926 07:34:40.319783 12770 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4da98cec4eeb435b822701ea535d1040 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:34:40.319782 12790 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bc11ca2eadd4b75b7ed372055a0fd56 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:34:40.325580 12785 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f07f342df5d4f609220ec3f3993d24b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4f07f342df5d4f609220ec3f3993d24b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f07f342df5d4f609220ec3f3993d24b" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 46639 } } peers { permanent_uuid: "4da98cec4eeb435b822701ea535d1040" member_type: VOTER last_known_addr { host: "127.22.53.253" port: 40235 } } peers { permanent_uuid: "1bc11ca2eadd4b75b7ed372055a0fd56" member_type: VOTER last_known_addr { host: "127.22.53.252" port: 39951 } } }
I20260926 07:34:40.325752 12785 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f07f342df5d4f609220ec3f3993d24b [sys.catalog]: This master's current role is: LEADER
I20260926 07:34:40.327131 12790 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bc11ca2eadd4b75b7ed372055a0fd56 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4f07f342df5d4f609220ec3f3993d24b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f07f342df5d4f609220ec3f3993d24b" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 46639 } } peers { permanent_uuid: "4da98cec4eeb435b822701ea535d1040" member_type: VOTER last_known_addr { host: "127.22.53.253" port: 40235 } } peers { permanent_uuid: "1bc11ca2eadd4b75b7ed372055a0fd56" member_type: VOTER last_known_addr { host: "127.22.53.252" port: 39951 } } }
I20260926 07:34:40.327263 12790 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bc11ca2eadd4b75b7ed372055a0fd56 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:34:40.327373 12793 catalog_manager.cc:1347] Generated new cluster ID: 1158b4cca83f4d97b540cc5b0decd8c5
I20260926 07:34:40.327440 12793 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:34:40.327863 12809 mvcc.cc:205] Tried to move back new op lower bound from 7333511496954048512 to 7333511496831352832. Current Snapshot: MvccSnapshot[applied={T|T < 7333511496954048512}]
I20260926 07:34:40.333503 12757 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f07f342df5d4f609220ec3f3993d24b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4f07f342df5d4f609220ec3f3993d24b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f07f342df5d4f609220ec3f3993d24b" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 46639 } } peers { permanent_uuid: "4da98cec4eeb435b822701ea535d1040" member_type: VOTER last_known_addr { host: "127.22.53.253" port: 40235 } } peers { permanent_uuid: "1bc11ca2eadd4b75b7ed372055a0fd56" member_type: VOTER last_known_addr { host: "127.22.53.252" port: 39951 } } }
I20260926 07:34:40.333666 12757 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f07f342df5d4f609220ec3f3993d24b [sys.catalog]: This master's current role is: LEADER
I20260926 07:34:40.335060 12770 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4da98cec4eeb435b822701ea535d1040 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4f07f342df5d4f609220ec3f3993d24b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f07f342df5d4f609220ec3f3993d24b" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 46639 } } peers { permanent_uuid: "4da98cec4eeb435b822701ea535d1040" member_type: VOTER last_known_addr { host: "127.22.53.253" port: 40235 } } peers { permanent_uuid: "1bc11ca2eadd4b75b7ed372055a0fd56" member_type: VOTER last_known_addr { host: "127.22.53.252" port: 39951 } } }
I20260926 07:34:40.335198 12770 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4da98cec4eeb435b822701ea535d1040 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:34:40.335399 12810 mvcc.cc:205] Tried to move back new op lower bound from 7333511496954048512 to 7333511496831352832. Current Snapshot: MvccSnapshot[applied={T|T < 7333511496954048512}]
I20260926 07:34:40.370406 12793 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:34:40.371785 12793 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:34:40.399269 12793 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4f07f342df5d4f609220ec3f3993d24b: Generated new TSK 0
I20260926 07:34:40.400307 12793 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 07:34:40.429790 12603 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:34:40.426625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46992 (local address 127.22.53.254:46639)
0926 07:34:40.426762 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:40.426767 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:40.428673 (+ 1906us) server_negotiation.cc:407] Connection header received
0926 07:34:40.428881 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:40.428884 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:40.428939 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:40.428993 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:40.429644 (+ 651us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46992: BlockingRecv error: recv got EOF from 127.0.0.1:46992 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260926 07:34:40.431195 12679 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:34:40.427396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34018 (local address 127.22.53.253:40235)
0926 07:34:40.427513 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:40.427518 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:40.430285 (+ 2767us) server_negotiation.cc:407] Connection header received
0926 07:34:40.430473 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:40.430477 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:40.430527 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:40.430577 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:40.431062 (+ 485us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34018: BlockingRecv error: recv got EOF from 127.0.0.1:34018 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260926 07:34:40.432771 12753 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:34:40.428088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60728 (local address 127.22.53.252:39951)
0926 07:34:40.428242 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:40.428246 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:40.431631 (+ 3385us) server_negotiation.cc:407] Connection header received
0926 07:34:40.432012 (+ 381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:40.432016 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:40.432068 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:40.432116 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:40.432653 (+ 537us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60728: BlockingRecv error: recv got EOF from 127.0.0.1:60728 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260926 07:34:40.439160 12679 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:34:40.437486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34032 (local address 127.22.53.253:40235)
0926 07:34:40.437603 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:40.437607 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:40.438085 (+ 478us) server_negotiation.cc:407] Connection header received
0926 07:34:40.438321 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:40.438325 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:40.438422 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:40.438478 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:40.439041 (+ 563us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34032: BlockingRecv error: recv got EOF from 127.0.0.1:34032 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260926 07:34:40.444516 12679 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:34:40.442919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34048 (local address 127.22.53.253:40235)
0926 07:34:40.443031 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:40.443035 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:40.443562 (+ 527us) server_negotiation.cc:407] Connection header received
0926 07:34:40.443754 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:40.443758 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:40.443811 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:40.443862 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:40.444392 (+ 530us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34048: BlockingRecv error: recv got EOF from 127.0.0.1:34048 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260926 07:34:40.450454 12679 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:34:40.449072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34052 (local address 127.22.53.253:40235)
0926 07:34:40.449208 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:40.449212 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:40.449489 (+ 277us) server_negotiation.cc:407] Connection header received
0926 07:34:40.449683 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:40.449686 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:40.449738 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:40.449790 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:40.450328 (+ 538us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34052: BlockingRecv error: recv got EOF from 127.0.0.1:34052 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260926 07:34:40.462150 12679 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:34:40.457216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34054 (local address 127.22.53.253:40235)
0926 07:34:40.457395 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:40.457400 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:40.461007 (+ 3607us) server_negotiation.cc:407] Connection header received
0926 07:34:40.461251 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:40.461255 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:40.461318 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:40.461391 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:40.462031 (+ 640us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34054: BlockingRecv error: recv got EOF from 127.0.0.1:34054 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
W20260926 07:34:40.474818 12679 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:34:40.472921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34066 (local address 127.22.53.253:40235)
0926 07:34:40.473049 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:40.473054 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:40.473658 (+ 604us) server_negotiation.cc:407] Connection header received
0926 07:34:40.473917 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:40.473922 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:40.473994 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:40.474061 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:40.474695 (+ 634us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34066: BlockingRecv error: recv got EOF from 127.0.0.1:34066 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260926 07:34:40.495869 12679 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:34:40.493930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34082 (local address 127.22.53.253:40235)
0926 07:34:40.494071 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:40.494076 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:40.494771 (+ 695us) server_negotiation.cc:407] Connection header received
0926 07:34:40.495011 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:40.495015 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:40.495075 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:40.495131 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:40.495745 (+ 614us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34082: BlockingRecv error: recv got EOF from 127.0.0.1:34082 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260926 07:34:40.532115 12679 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:34:40.530791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34098 (local address 127.22.53.253:40235)
0926 07:34:40.530942 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:40.530947 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:40.531142 (+ 195us) server_negotiation.cc:407] Connection header received
0926 07:34:40.531343 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:40.531347 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:40.531398 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:40.531494 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:40.531963 (+ 469us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34098: BlockingRecv error: recv got EOF from 127.0.0.1:34098 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260926 07:34:40.599701 12679 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:34:40.598596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34108 (local address 127.22.53.253:40235)
0926 07:34:40.598790 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:40.598794 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:40.598812 (+ 18us) server_negotiation.cc:407] Connection header received
0926 07:34:40.598924 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:40.598927 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:40.598975 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:40.599050 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:40.599559 (+ 509us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34108: BlockingRecv error: recv got EOF from 127.0.0.1:34108 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
W20260926 07:34:40.731231 12679 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:34:40.729963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34112 (local address 127.22.53.253:40235)
0926 07:34:40.730158 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:40.730163 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:40.730281 (+ 118us) server_negotiation.cc:407] Connection header received
0926 07:34:40.730468 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:40.730472 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:40.730529 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:40.730622 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:40.731087 (+ 465us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34112: BlockingRecv error: recv got EOF from 127.0.0.1:34112 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
W20260926 07:34:40.991109 12679 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:34:40.989860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34114 (local address 127.22.53.253:40235)
0926 07:34:40.990015 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:40.990020 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:40.990141 (+ 121us) server_negotiation.cc:407] Connection header received
0926 07:34:40.990342 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:40.990345 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:40.990403 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:40.990495 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:40.990972 (+ 477us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34114: BlockingRecv error: recv got EOF from 127.0.0.1:34114 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 07:34:41.291210 12782 catalog_manager.cc:1259] Loaded cluster ID: 1158b4cca83f4d97b540cc5b0decd8c5
I20260926 07:34:41.291451 12782 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4da98cec4eeb435b822701ea535d1040: loading cluster ID for follower catalog manager: success
I20260926 07:34:41.293295 12782 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4da98cec4eeb435b822701ea535d1040: acquiring CA information for follower catalog manager: success
I20260926 07:34:41.294636 12782 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4da98cec4eeb435b822701ea535d1040: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 07:34:41.310672 12805 catalog_manager.cc:1259] Loaded cluster ID: 1158b4cca83f4d97b540cc5b0decd8c5
I20260926 07:34:41.310789 12805 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1bc11ca2eadd4b75b7ed372055a0fd56: loading cluster ID for follower catalog manager: success
I20260926 07:34:41.312677 12805 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1bc11ca2eadd4b75b7ed372055a0fd56: acquiring CA information for follower catalog manager: success
I20260926 07:34:41.314067 12805 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1bc11ca2eadd4b75b7ed372055a0fd56: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 07:34:41.509466 12881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:41.505467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34124 (local address 127.22.53.253:40235)
0926 07:34:41.505713 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:41.505716 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:41.505733 (+ 17us) server_negotiation.cc:407] Connection header received
0926 07:34:41.505787 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:41.505790 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:41.505913 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:41.506025 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:34:41.506339 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:41.506873 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:41.507784 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:41.508030 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:41.508113 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:41.508480 (+ 367us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:34:41.509024 (+ 544us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:34:41.509130 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:41.509260 (+ 130us) server_negotiation.cc:299] Negotiation successful
0926 07:34:41.509336 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":85,"threads_started":1}
I20260926 07:34:41.516889 12887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:41.513067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60736 (local address 127.22.53.252:39951)
0926 07:34:41.513360 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:41.513363 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:41.513378 (+ 15us) server_negotiation.cc:407] Connection header received
0926 07:34:41.513522 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:41.513524 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:41.513645 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:41.513756 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:34:41.513944 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:41.514465 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:41.515282 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:41.515508 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:41.515591 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:41.515896 (+ 305us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:34:41.516368 (+ 472us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:34:41.516479 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:41.516602 (+ 123us) server_negotiation.cc:299] Negotiation successful
0926 07:34:41.516682 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":95,"threads_started":1}
I20260926 07:34:41.524726 12881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:41.520490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34126 (local address 127.22.53.253:40235)
0926 07:34:41.520657 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:41.520662 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:41.520817 (+ 155us) server_negotiation.cc:407] Connection header received
0926 07:34:41.521031 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:41.521036 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:41.521097 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:41.521166 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:34:41.521547 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:41.522095 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:41.523444 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:41.523794 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:41.523919 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:41.524107 (+ 188us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:34:41.524215 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:34:41.524280 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:41.524443 (+ 163us) server_negotiation.cc:299] Negotiation successful
0926 07:34:41.524503 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260926 07:34:41.525157 12887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:41.520254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60750 (local address 127.22.53.252:39951)
0926 07:34:41.520385 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:41.520391 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:41.520649 (+ 258us) server_negotiation.cc:407] Connection header received
0926 07:34:41.520895 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:41.520900 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:41.520991 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:41.521072 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:34:41.521327 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:41.522175 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:41.523795 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:41.524160 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:41.524279 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:41.524522 (+ 243us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:34:41.524631 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:34:41.524681 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:41.524968 (+ 287us) server_negotiation.cc:299] Negotiation successful
0926 07:34:41.525027 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 07:34:41.525422 12894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:41.520113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46998 (local address 127.22.53.254:46639)
0926 07:34:41.520701 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:41.520708 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:41.520808 (+ 100us) server_negotiation.cc:407] Connection header received
0926 07:34:41.521022 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:41.521025 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:41.521220 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:41.521338 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:34:41.521606 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:41.522397 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:41.523254 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:41.523581 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:41.523690 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:41.524178 (+ 488us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:34:41.524604 (+ 426us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:34:41.524702 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:41.525153 (+ 451us) server_negotiation.cc:299] Negotiation successful
0926 07:34:41.525258 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":499,"thread_start_us":136,"threads_started":1}
I20260926 07:34:41.529313 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 12532
I20260926 07:34:41.536171 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 12607
I20260926 07:34:41.542873 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 12681
2026-09-26T07:34:41Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3167 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407783529717-22743-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:34:41 dist-test-slave-wdjc krb5kdc[12898](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:34:41 dist-test-slave-wdjc krb5kdc[12898](info): set up 2 sockets
Sep 26 07:34:41 dist-test-slave-wdjc krb5kdc[12898](info): commencing operation
krb5kdc: starting...
W20260926 07:34:42.586050 22743 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407783529717-22743-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407783529717-22743-0/krb5kdc/test-user.keytab.
Sep 26 07:34:42 dist-test-slave-wdjc krb5kdc[12898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408082, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:34:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:34:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.53.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.53.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.53.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab.
Entry for principal kudu/127.22.53.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab.
I20260926 07:34:42.720048 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407783529717-22743-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407783529717-22743-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407783529717-22743-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.53.254:34245
--webserver_interface=127.22.53.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab
--principal=kudu/127.22.53.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:37681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.53.254:34245,127.22.53.253:46247,127.22.53.252:33181
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:34:42.838770 12911 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 07:34:42.839190 12911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:34:42.839280 12911 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:34:42.839367 12911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:34:42.843713 12911 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:34:42.843849 12911 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:34:42.843886 12911 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:34:42.843926 12911 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:34:42.843961 12911 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:34:42.849891 12911 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:37681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407783529717-22743-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407783529717-22743-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.53.254:34245,127.22.53.253:46247,127.22.53.252:33181
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.254.keytab
--principal=kudu/127.22.53.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.53.254:34245
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.53.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12911
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407783529717-22743-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:34:42.851190 12911 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:34:42.852344 12911 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:34:42.861829 12916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:42.861824 12919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:42.861824 12917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:34:42.861936 12911 server_base.cc:1034] running on GCE node
I20260926 07:34:42.862841 12911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:34:42.864091 12911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:34:42.865334 12911 hybrid_clock.cc:648] HybridClock initialized: now 1790408082865299 us; error 51 us; skew 500 ppm
Sep 26 07:34:42 dist-test-slave-wdjc krb5kdc[12898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408082, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:34:42.873476 12911 init.cc:377] Logged in from keytab as kudu/127.22.53.254@KRBTEST.COM (short username kudu)
I20260926 07:34:42.874799 12911 webserver.cc:460] Webserver started at http://127.22.53.254:42895/ using document root <none> and password file <none>
I20260926 07:34:42.875465 12911 fs_manager.cc:359] Metadata directory not provided
I20260926 07:34:42.875572 12911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:34:42.875835 12911 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:34:42.878211 12911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407783529717-22743-0/minicluster-data/master-0/data/instance:
uuid: "39bf0f403cb7450eac133a20764b55c3"
format_stamp: "Formatted at 2026-09-26 07:34:42 on dist-test-slave-wdjc"
server_key: "12bd00e7a97936712c5fb75e1f3e12a6"
server_key_iv: "7db02ca9b3948b799529051aaff33b2e"
server_key_version: "encryptionkey@0"
I20260926 07:34:42.878819 12911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407783529717-22743-0/minicluster-data/master-0/wal/instance:
uuid: "39bf0f403cb7450eac133a20764b55c3"
format_stamp: "Formatted at 2026-09-26 07:34:42 on dist-test-slave-wdjc"
server_key: "12bd00e7a97936712c5fb75e1f3e12a6"
server_key_iv: "7db02ca9b3948b799529051aaff33b2e"
server_key_version: "encryptionkey@0"
I20260926 07:34:42.882985 12911 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 07:34:42.885958 12926 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:42.887465 12911 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:34:42.887635 12911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407783529717-22743-0/minicluster-data/master-0/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407783529717-22743-0/minicluster-data/master-0/wal
uuid: "39bf0f403cb7450eac133a20764b55c3"
format_stamp: "Formatted at 2026-09-26 07:34:42 on dist-test-slave-wdjc"
server_key: "12bd00e7a97936712c5fb75e1f3e12a6"
server_key_iv: "7db02ca9b3948b799529051aaff33b2e"
server_key_version: "encryptionkey@0"
I20260926 07:34:42.887776 12911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407783529717-22743-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407783529717-22743-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407783529717-22743-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:34:42.923491 12911 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:34:42.927547 12911 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:34:42.927814 12911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:34:42.935788 12911 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.254:34245
I20260926 07:34:42.936112 12978 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.254:34245 every 8 connection(s)
I20260926 07:34:42.936996 12911 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407783529717-22743-0/minicluster-data/master-0/data/info.pb
I20260926 07:34:42.938215 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 12911
I20260926 07:34:42.938352 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407783529717-22743-0/minicluster-data/master-0/wal/instance
I20260926 07:34:42.938688 22743 external_mini_cluster.cc:1428] Setting key 38972acd83531c5b06759d743514388c
I20260926 07:34:42.941114 12979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:34:42.945566 12979 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.53.254" port: 34245 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.22.53.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.53.253@KRBTEST.COM" created.
Sep 26 07:34:42 dist-test-slave-wdjc krb5kdc[12898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408082, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.254@KRBTEST.COM for kudu/127.22.53.254@KRBTEST.COM
I20260926 07:34:42.965920 12983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:42.948422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:34245 (local address 127.0.0.1:42536)
0926 07:34:42.949185 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:42.949235 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:42.950587 (+ 1352us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:42.951651 (+ 1064us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:42.951658 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:42.952246 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:42.952760 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:42.952789 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:42.954161 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:42.954165 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:42.955691 (+ 1526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:42.955713 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:42.956624 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:42.956629 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:42.956802 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:42.957593 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:42.957619 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:42.959576 (+ 1957us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:34:42.962229 (+ 2653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:42.962241 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:42.962261 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:42.962542 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:42.962981 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:42.962990 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:42.962991 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:42.963180 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:42.963998 (+ 818us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:34:42.964004 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:34:42.964181 (+ 177us) client_negotiation.cc:769] Sending connection context
0926 07:34:42.964410 (+ 229us) client_negotiation.cc:240] Negotiation successful
0926 07:34:42.965021 (+ 611us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"mutex_wait_us":173,"spinlock_wait_cycles":207616,"thread_start_us":279,"threads_started":1}
I20260926 07:34:42.965920 12982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:42.948422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42536 (local address 127.22.53.254:34245)
0926 07:34:42.949098 (+ 676us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:42.949118 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:42.949372 (+ 254us) server_negotiation.cc:407] Connection header received
0926 07:34:42.950995 (+ 1623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:42.951005 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:42.951350 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:42.951581 (+ 231us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:42.953004 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:42.953988 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:42.955893 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:42.956360 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:42.956511 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:42.959807 (+ 3296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:42.959816 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:42.959831 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:42.959868 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:42.962050 (+ 2182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:42.962681 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:42.962700 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:42.962701 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:42.962830 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:42.963319 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:42.963322 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:42.963323 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:42.963850 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:42.963968 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:42.964525 (+ 557us) server_negotiation.cc:299] Negotiation successful
0926 07:34:42.964943 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":46,"server-negotiator.queue_time_us":507,"thread_start_us":219,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.53.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.253.keytab.
Entry for principal kudu/127.22.53.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.253.keytab.
I20260926 07:34:42.968619 12979 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.53.253" port: 46247 } has no permanent_uuid. Determining permanent_uuid...
I20260926 07:34:42.969627 12983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:42.969273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.253:46247 (local address 127.0.0.1:39024)
0926 07:34:42.969449 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:42.969535 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.253:46247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260926 07:34:42.969622 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407783529717-22743-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407783529717-22743-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407783529717-22743-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407783529717-22743-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.53.253:46247
--webserver_interface=127.22.53.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.253.keytab
--principal=kudu/127.22.53.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:37681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.53.254:34245,127.22.53.253:46247,127.22.53.252:33181
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:34:42.969921 12930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.53.253:46247: connect: Connection refused (error 111)
I20260926 07:34:42.970949 12983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:42.970691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.253:46247 (local address 127.0.0.1:39036)
0926 07:34:42.970810 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:42.970869 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.253:46247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260926 07:34:42.971146 12979 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.53.253:46247: Network error: Client connection negotiation failed: client connection to 127.22.53.253:46247: connect: Connection refused (error 111)
I20260926 07:34:43.020479 12979 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.53.253" port: 46247 } attempt: 1
I20260926 07:34:43.021615 12983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:43.021256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.253:46247 (local address 127.0.0.1:39048)
0926 07:34:43.021457 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:43.021528 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.253:46247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260926 07:34:43.022490 12983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:43.022234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.253:46247 (local address 127.0.0.1:39060)
0926 07:34:43.022372 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:43.022424 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.253:46247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260926 07:34:43.022732 12979 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.53.253:46247: Network error: Client connection negotiation failed: client connection to 127.22.53.253:46247: connect: Connection refused (error 111)
W20260926 07:34:43.088750 12986 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 07:34:43.089088 12986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:34:43.089113 12986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:34:43.089205 12986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260926 07:34:43.090956 12979 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.53.253" port: 46247 } attempt: 2
I20260926 07:34:43.092007 12983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:43.091685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.253:46247 (local address 127.0.0.1:39070)
0926 07:34:43.091864 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:43.091930 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.253:46247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260926 07:34:43.092922 12983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:43.092647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.253:46247 (local address 127.0.0.1:39076)
0926 07:34:43.092789 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:43.092841 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.253:46247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260926 07:34:43.093087 12979 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.53.253:46247: Network error: Client connection negotiation failed: client connection to 127.22.53.253:46247: connect: Connection refused (error 111)
W20260926 07:34:43.094347 12986 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:34:43.094409 12986 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:34:43.094429 12986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:34:43.094448 12986 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:34:43.094465 12986 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:34:43.100390 12986 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:37681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407783529717-22743-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407783529717-22743-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.22.53.254:34245,127.22.53.253:46247,127.22.53.252:33181
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.253.keytab
--principal=kudu/127.22.53.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.53.253:46247
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407783529717-22743-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.22.53.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12986
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407783529717-22743-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:34:43.101625 12986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:34:43.102713 12986 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:34:43.111867 12994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:43.111868 12991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:34:43.111979 12986 server_base.cc:1034] running on GCE node
W20260926 07:34:43.111863 12992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:34:43.112821 12986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:34:43.114030 12986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:34:43.115206 12986 hybrid_clock.cc:648] HybridClock initialized: now 1790408083115189 us; error 37 us; skew 500 ppm
Sep 26 07:34:43 dist-test-slave-wdjc krb5kdc[12898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408083, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:34:43.118443 12986 init.cc:377] Logged in from keytab as kudu/127.22.53.253@KRBTEST.COM (short username kudu)
I20260926 07:34:43.119481 12986 webserver.cc:460] Webserver started at http://127.22.53.253:39739/ using document root <none> and password file <none>
I20260926 07:34:43.120087 12986 fs_manager.cc:359] Metadata directory not provided
I20260926 07:34:43.120153 12986 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:34:43.120344 12986 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:34:43.122691 12986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407783529717-22743-0/minicluster-data/master-1/data/instance:
uuid: "07d8da00a0164cbe87f7570ab2a0361d"
format_stamp: "Formatted at 2026-09-26 07:34:43 on dist-test-slave-wdjc"
server_key: "06e05af61fb819477779e9bbfe64a06f"
server_key_iv: "829f72aab03f0a7aa5508de54dd6b13c"
server_key_version: "encryptionkey@0"
I20260926 07:34:43.123291 12986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407783529717-22743-0/minicluster-data/master-1/wal/instance:
uuid: "07d8da00a0164cbe87f7570ab2a0361d"
format_stamp: "Formatted at 2026-09-26 07:34:43 on dist-test-slave-wdjc"
server_key: "06e05af61fb819477779e9bbfe64a06f"
server_key_iv: "829f72aab03f0a7aa5508de54dd6b13c"
server_key_version: "encryptionkey@0"
I20260926 07:34:43.127658 12986 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 07:34:43.130568 13001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:43.132195 12986 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 07:34:43.132347 12986 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407783529717-22743-0/minicluster-data/master-1/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407783529717-22743-0/minicluster-data/master-1/wal
uuid: "07d8da00a0164cbe87f7570ab2a0361d"
format_stamp: "Formatted at 2026-09-26 07:34:43 on dist-test-slave-wdjc"
server_key: "06e05af61fb819477779e9bbfe64a06f"
server_key_iv: "829f72aab03f0a7aa5508de54dd6b13c"
server_key_version: "encryptionkey@0"
I20260926 07:34:43.132476 12986 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407783529717-22743-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407783529717-22743-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407783529717-22743-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:34:43.148494 12986 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:34:43.152791 12986 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:34:43.153007 12986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:34:43.161832 12986 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.253:46247
I20260926 07:34:43.162145 13053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.253:46247 every 8 connection(s)
I20260926 07:34:43.163166 12986 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407783529717-22743-0/minicluster-data/master-1/data/info.pb
I20260926 07:34:43.166407 13054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:34:43.167385 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 12986
I20260926 07:34:43.167497 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407783529717-22743-0/minicluster-data/master-1/wal/instance
I20260926 07:34:43.167845 22743 external_mini_cluster.cc:1428] Setting key 2cca70dc3592336d5d53c391d44e8a45
I20260926 07:34:43.171779 13054 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.53.254" port: 34245 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.22.53.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.53.252@KRBTEST.COM" created.
I20260926 07:34:43.184288 12979 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.53.253" port: 46247 } attempt: 3
Sep 26 07:34:43 dist-test-slave-wdjc krb5kdc[12898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408083, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.253@KRBTEST.COM for kudu/127.22.53.254@KRBTEST.COM
Sep 26 07:34:43 dist-test-slave-wdjc krb5kdc[12898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408082, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.254@KRBTEST.COM for kudu/127.22.53.253@KRBTEST.COM
I20260926 07:34:43.192091 12982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:43.174754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42548 (local address 127.22.53.254:34245)
0926 07:34:43.174956 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:43.174961 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:43.175940 (+ 979us) server_negotiation.cc:407] Connection header received
0926 07:34:43.177139 (+ 1199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:43.177142 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:43.177205 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:43.177306 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:43.180117 (+ 2811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:43.180787 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:43.182516 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:43.182959 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:43.183060 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:43.186101 (+ 3041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:43.186107 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:43.186110 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:43.186143 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:43.189266 (+ 3123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:43.189935 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:43.189939 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:43.189940 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:43.190078 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:43.190545 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:43.190549 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:43.190551 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:43.190783 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:43.190912 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:43.191720 (+ 808us) server_negotiation.cc:299] Negotiation successful
0926 07:34:43.191877 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 07:34:43.193228 13057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:43.175153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:34245 (local address 127.0.0.1:42548)
0926 07:34:43.175733 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:43.175793 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:43.176903 (+ 1110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:43.177847 (+ 944us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:43.177867 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:43.178619 (+ 752us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:43.179882 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:43.179911 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:43.180969 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:43.180976 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:43.182339 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:43.182352 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:43.183124 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:43.183127 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:43.183300 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:43.183993 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:43.184011 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:43.185912 (+ 1901us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:34:43.189421 (+ 3509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:43.189427 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:43.189443 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:43.189804 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:43.190213 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:43.190219 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:43.190224 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:43.190462 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:43.190947 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:34:43.190954 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:34:43.191404 (+ 450us) client_negotiation.cc:769] Sending connection context
0926 07:34:43.191644 (+ 240us) client_negotiation.cc:240] Negotiation successful
0926 07:34:43.192019 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":184,"threads_started":1}
I20260926 07:34:43.195047 13054 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.53.253" port: 46247 } has no permanent_uuid. Determining permanent_uuid...
I20260926 07:34:43.196731 12983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:43.184879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.253:46247 (local address 127.0.0.1:39084)
0926 07:34:43.185117 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:43.185135 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:43.185226 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:43.186155 (+ 929us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:43.186159 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:43.186393 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:43.186540 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:43.186547 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:43.188084 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:43.188087 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:43.188802 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:43.188815 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:43.189710 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:43.189714 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:43.189853 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:43.190423 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:43.190439 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:43.192290 (+ 1851us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:34:43.194759 (+ 2469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:43.194763 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:43.194765 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:43.195072 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:43.195421 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:43.195424 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:43.195425 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:43.195534 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:43.196150 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:34:43.196155 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:34:43.196270 (+ 115us) client_negotiation.cc:769] Sending connection context
0926 07:34:43.196380 (+ 110us) client_negotiation.cc:240] Negotiation successful
0926 07:34:43.196593 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144}
I20260926 07:34:43.197063 13059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:43.185041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39084 (local address 127.22.53.253:46247)
0926 07:34:43.185566 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:43.185579 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:43.185609 (+ 30us) server_negotiation.cc:407] Connection header received
0926 07:34:43.185673 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:43.185684 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:43.185954 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:43.186177 (+ 223us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:43.186664 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:43.187953 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:43.188957 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:43.189571 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:43.189671 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:43.192420 (+ 2749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:43.192429 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:43.192448 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:43.192620 (+ 172us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:43.194636 (+ 2016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:43.195190 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:43.195196 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:43.195198 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:43.195289 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:43.195641 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:43.195646 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:43.195648 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:43.196044 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:43.196146 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:43.196477 (+ 331us) server_negotiation.cc:299] Negotiation successful
0926 07:34:43.196663 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":135,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.53.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.252.keytab.
Entry for principal kudu/127.22.53.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.252.keytab.
I20260926 07:34:43.199165 12979 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.53.252" port: 33181 } has no permanent_uuid. Determining permanent_uuid...
I20260926 07:34:43.199823 22743 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
/tmp/dist-test-taskdIE4X7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407783529717-22743-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407783529717-22743-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407783529717-22743-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407783529717-22743-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.53.252:33181
--webserver_interface=127.22.53.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.252.keytab
--principal=kudu/127.22.53.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.53.212:37681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.53.254:34245,127.22.53.253:46247,127.22.53.252:33181
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407783529717-22743-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407783529717-22743-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407783529717-22743-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:34:43.200057 12983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:43.199713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.252:33181 (local address 127.0.0.1:36476)
0926 07:34:43.199885 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:43.199956 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.252:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
Sep 26 07:34:43 dist-test-slave-wdjc krb5kdc[12898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408083, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.253@KRBTEST.COM for kudu/127.22.53.253@KRBTEST.COM
I20260926 07:34:43.201176 13061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:43.200563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.252:33181 (local address 127.0.0.1:36492)
0926 07:34:43.201011 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:43.201068 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.252:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":134,"threads_started":1}
W20260926 07:34:43.201313 12979 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.53.252:33181: Network error: Client connection negotiation failed: client connection to 127.22.53.252:33181: connect: Connection refused (error 111)
I20260926 07:34:43.205570 13057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:43.195628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.253:46247 (local address 127.0.0.1:39092)
0926 07:34:43.195762 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:43.195776 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:43.195852 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:43.196345 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:43.196347 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:43.196539 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:43.196681 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:43.196688 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:43.197788 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:43.197791 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:43.198446 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:43.198459 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:43.199141 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:43.199144 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:43.199237 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:43.199940 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:43.199961 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:43.201599 (+ 1638us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:34:43.204013 (+ 2414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:43.204017 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:43.204019 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:43.204219 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:43.204499 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:43.204504 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:43.204506 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:43.204614 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:43.205008 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:34:43.205011 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:34:43.205096 (+ 85us) client_negotiation.cc:769] Sending connection context
0926 07:34:43.205202 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 07:34:43.205370 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260926 07:34:43.205582 13060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:43.195703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39092 (local address 127.22.53.253:46247)
0926 07:34:43.196117 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:43.196128 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:43.196140 (+ 12us) server_negotiation.cc:407] Connection header received
0926 07:34:43.196171 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:43.196173 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:43.196217 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:43.196321 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:43.196948 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:43.197683 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:43.198568 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:43.198921 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:43.199015 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:43.201763 (+ 2748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:43.201769 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:43.201772 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:43.201797 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:43.203887 (+ 2090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:43.204323 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:43.204327 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:43.204329 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:43.204410 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:43.204703 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:43.204708 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:43.204710 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:43.204912 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:43.205022 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:43.205258 (+ 236us) server_negotiation.cc:299] Negotiation successful
0926 07:34:43.205433 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":338,"thread_start_us":198,"threads_started":1}
I20260926 07:34:43.206360 13054 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.53.252" port: 33181 } has no permanent_uuid. Determining permanent_uuid...
I20260926 07:34:43.207281 13057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:43.206893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.252:33181 (local address 127.0.0.1:36498)
0926 07:34:43.207059 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:43.207177 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.252:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260926 07:34:43.207624 13003 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.53.252:33181: connect: Connection refused (error 111)
I20260926 07:34:43.208788 13057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:43.208394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.252:33181 (local address 127.0.0.1:36506)
0926 07:34:43.208618 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:43.208685 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.252:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260926 07:34:43.209003 13054 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.53.252:33181: Network error: Client connection negotiation failed: client connection to 127.22.53.252:33181: connect: Connection refused (error 111)
I20260926 07:34:43.232515 12979 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.53.252" port: 33181 } attempt: 1
I20260926 07:34:43.233511 13061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:43.233189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.252:33181 (local address 127.0.0.1:36508)
0926 07:34:43.233357 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:43.233430 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.252:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260926 07:34:43.234496 13061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:43.234188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.252:33181 (local address 127.0.0.1:36520)
0926 07:34:43.234350 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:43.234411 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.252:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
W20260926 07:34:43.234688 12979 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.53.252:33181: Network error: Client connection negotiation failed: client connection to 127.22.53.252:33181: connect: Connection refused (error 111)
I20260926 07:34:43.258369 13054 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.53.252" port: 33181 } attempt: 1
I20260926 07:34:43.259322 13057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:43.259013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.252:33181 (local address 127.0.0.1:36526)
0926 07:34:43.259171 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:43.259235 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.252:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260926 07:34:43.260089 13057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:43.259866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.252:33181 (local address 127.0.0.1:36536)
0926 07:34:43.259979 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:43.260032 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.252:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260926 07:34:43.260241 13054 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.53.252:33181: Network error: Client connection negotiation failed: client connection to 127.22.53.252:33181: connect: Connection refused (error 111)
I20260926 07:34:43.309891 12979 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.53.252" port: 33181 } attempt: 2
I20260926 07:34:43.310885 13061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:43.310555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.252:33181 (local address 127.0.0.1:36544)
0926 07:34:43.310733 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:43.310802 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.252:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260926 07:34:43.311712 13061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:43.311473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.252:33181 (local address 127.0.0.1:36558)
0926 07:34:43.311591 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:43.311646 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.252:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260926 07:34:43.311854 12979 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.53.252:33181: Network error: Client connection negotiation failed: client connection to 127.22.53.252:33181: connect: Connection refused (error 111)
W20260926 07:34:43.321975 13062 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 07:34:43.322386 13062 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:34:43.322453 13062 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:34:43.322557 13062 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:34:43.326977 13062 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:34:43.327090 13062 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:34:43.327127 13062 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:34:43.327164 13062 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:34:43.327219 13062 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:34:43.328428 13054 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.53.252" port: 33181 } attempt: 2
I20260926 07:34:43.329614 13057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:43.329262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.252:33181 (local address 127.0.0.1:36562)
0926 07:34:43.329441 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:43.329525 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.252:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260926 07:34:43.330469 13057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:43.330208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.252:33181 (local address 127.0.0.1:36576)
0926 07:34:43.330347 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:43.330408 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.53.252:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260926 07:34:43.330626 13054 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.53.252:33181: Network error: Client connection negotiation failed: client connection to 127.22.53.252:33181: connect: Connection refused (error 111)
I20260926 07:34:43.333604 13062 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.53.212:37681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407783529717-22743-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407783529717-22743-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.22.53.254:34245,127.22.53.253:46247,127.22.53.252:33181
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407783529717-22743-0/krb5kdc/kudu_127.22.53.252.keytab
--principal=kudu/127.22.53.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.53.252:33181
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407783529717-22743-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.22.53.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13062
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407783529717-22743-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:34:43.334823 13062 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:34:43.335944 13062 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:34:43.345346 13071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:43.345386 13068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:34:43.345386 13069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:34:43.345846 13062 server_base.cc:1034] running on GCE node
I20260926 07:34:43.346525 13062 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:34:43.347843 13062 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:34:43.349066 13062 hybrid_clock.cc:648] HybridClock initialized: now 1790408083349046 us; error 37 us; skew 500 ppm
Sep 26 07:34:43 dist-test-slave-wdjc krb5kdc[12898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408083, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:34:43.352135 13062 init.cc:377] Logged in from keytab as kudu/127.22.53.252@KRBTEST.COM (short username kudu)
I20260926 07:34:43.353398 13062 webserver.cc:460] Webserver started at http://127.22.53.252:41717/ using document root <none> and password file <none>
I20260926 07:34:43.354048 13062 fs_manager.cc:359] Metadata directory not provided
I20260926 07:34:43.354158 13062 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:34:43.354445 13062 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:34:43.357514 13062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407783529717-22743-0/minicluster-data/master-2/data/instance:
uuid: "cc65889b2a1145d3bad180231dc13a85"
format_stamp: "Formatted at 2026-09-26 07:34:43 on dist-test-slave-wdjc"
server_key: "4d49e750ccffc2ef45778cfb2846224b"
server_key_iv: "b512d3f7f6cf39e34cb9f8d37cbd7dab"
server_key_version: "encryptionkey@0"
I20260926 07:34:43.358219 13062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407783529717-22743-0/minicluster-data/master-2/wal/instance:
uuid: "cc65889b2a1145d3bad180231dc13a85"
format_stamp: "Formatted at 2026-09-26 07:34:43 on dist-test-slave-wdjc"
server_key: "4d49e750ccffc2ef45778cfb2846224b"
server_key_iv: "b512d3f7f6cf39e34cb9f8d37cbd7dab"
server_key_version: "encryptionkey@0"
I20260926 07:34:43.362891 13062 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 07:34:43.366412 13078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:34:43.368897 13062 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 07:34:43.369103 13062 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407783529717-22743-0/minicluster-data/master-2/data,/tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407783529717-22743-0/minicluster-data/master-2/wal
uuid: "cc65889b2a1145d3bad180231dc13a85"
format_stamp: "Formatted at 2026-09-26 07:34:43 on dist-test-slave-wdjc"
server_key: "4d49e750ccffc2ef45778cfb2846224b"
server_key_iv: "b512d3f7f6cf39e34cb9f8d37cbd7dab"
server_key_version: "encryptionkey@0"
I20260926 07:34:43.369275 13062 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407783529717-22743-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407783529717-22743-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407783529717-22743-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:34:43.381522 13062 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:34:43.385363 13062 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:34:43.385599 13062 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:34:43.393677 13062 rpc_server.cc:307] RPC server started. Bound to: 127.22.53.252:33181
I20260926 07:34:43.393976 13130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.53.252:33181 every 8 connection(s)
I20260926 07:34:43.394802 13062 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407783529717-22743-0/minicluster-data/master-2/data/info.pb
I20260926 07:34:43.397902 13131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:34:43.398077 22743 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu as pid 13062
I20260926 07:34:43.398202 22743 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdIE4X7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407783529717-22743-0/minicluster-data/master-2/wal/instance
I20260926 07:34:43.398574 22743 external_mini_cluster.cc:1428] Setting key 6763cd7ae6d5e8c56f5da6d1026c0861
I20260926 07:34:43.403398 13131 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.53.254" port: 34245 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 07:34:43 dist-test-slave-wdjc krb5kdc[12898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408082, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.53.252@KRBTEST.COM
I20260926 07:34:43.411042 12979 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.53.252" port: 33181 } attempt: 3
Sep 26 07:34:43 dist-test-slave-wdjc krb5kdc[12898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408083, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.252@KRBTEST.COM for kudu/127.22.53.254@KRBTEST.COM
I20260926 07:34:43.416620 12982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:43.405577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59292 (local address 127.22.53.254:34245)
0926 07:34:43.405718 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:43.405721 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:43.406131 (+ 410us) server_negotiation.cc:407] Connection header received
0926 07:34:43.406303 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:43.406306 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:43.406362 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:43.406457 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:43.407166 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:43.407663 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:43.408596 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:43.408962 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:43.409101 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:43.412345 (+ 3244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:43.412355 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:43.412358 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:43.412384 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:43.414400 (+ 2016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:43.414997 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:43.415001 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:43.415002 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:43.415112 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:43.415462 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:43.415465 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:43.415467 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:43.415659 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:43.415780 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:43.416261 (+ 481us) server_negotiation.cc:299] Negotiation successful
0926 07:34:43.416399 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 07:34:43.416952 13135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:43.405532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.254:34245 (local address 127.0.0.1:59292)
0926 07:34:43.406004 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:43.406048 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:43.406200 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:43.406512 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:43.406519 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:43.406913 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:43.407039 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:43.407044 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:43.407804 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:43.407808 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:43.408445 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:43.408453 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:43.409187 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:43.409191 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:43.409322 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:43.410067 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:43.410089 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:43.412182 (+ 2093us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:34:43.414511 (+ 2329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:43.414522 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:43.414528 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:43.414877 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:43.415238 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:43.415243 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:43.415246 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:43.415344 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:43.415838 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:34:43.415846 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:34:43.415979 (+ 133us) client_negotiation.cc:769] Sending connection context
0926 07:34:43.416141 (+ 162us) client_negotiation.cc:240] Negotiation successful
0926 07:34:43.416393 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":171,"threads_started":1}
I20260926 07:34:43.417186 13134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:43.400132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36586 (local address 127.22.53.252:33181)
0926 07:34:43.400708 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:43.400724 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:43.400764 (+ 40us) server_negotiation.cc:407] Connection header received
0926 07:34:43.401547 (+ 783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:43.401568 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:43.401986 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:43.402296 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:43.403285 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:43.404419 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:43.405548 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:43.406280 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:43.406520 (+ 240us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:43.409214 (+ 2694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:43.409221 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:43.409235 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:43.409272 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:43.412622 (+ 3350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:43.413320 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:43.413327 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:43.413335 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:43.413491 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:43.413889 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:43.413893 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:43.413895 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:43.414459 (+ 564us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:43.414736 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:43.415079 (+ 343us) server_negotiation.cc:299] Negotiation successful
0926 07:34:43.415433 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":147,"threads_started":1}
Sep 26 07:34:43 dist-test-slave-wdjc krb5kdc[12898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408082, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.254@KRBTEST.COM for kudu/127.22.53.252@KRBTEST.COM
I20260926 07:34:43.418536 13131 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.53.253" port: 46247 } has no permanent_uuid. Determining permanent_uuid...
I20260926 07:34:43.421790 13054 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.53.252" port: 33181 } attempt: 3
I20260926 07:34:43.422868 13061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:43.411615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.252:33181 (local address 127.0.0.1:36588)
0926 07:34:43.411862 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:43.411878 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:43.411984 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:43.412724 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:43.412728 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:43.412961 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:43.413114 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:43.413125 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:43.414321 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:43.414325 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:43.415184 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:43.415197 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:43.415804 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:43.415808 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:43.416104 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:43.416668 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:43.416683 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:43.418298 (+ 1615us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:34:43.421058 (+ 2760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:43.421062 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:43.421064 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:43.421287 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:43.421603 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:43.421605 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:43.421607 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:43.421700 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:43.422252 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:34:43.422255 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:34:43.422363 (+ 108us) client_negotiation.cc:769] Sending connection context
0926 07:34:43.422485 (+ 122us) client_negotiation.cc:240] Negotiation successful
0926 07:34:43.422645 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147}
I20260926 07:34:43.423727 13136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:43.411824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36588 (local address 127.22.53.252:33181)
0926 07:34:43.412364 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:43.412373 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:43.412389 (+ 16us) server_negotiation.cc:407] Connection header received
0926 07:34:43.412433 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:43.412438 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:43.412515 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:43.412640 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:43.413260 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:43.414149 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:43.415341 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:43.415642 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:43.415750 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:43.418461 (+ 2711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:43.418465 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:43.418467 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:43.418495 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:43.420947 (+ 2452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:43.421402 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:43.421406 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:43.421408 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:43.421498 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:43.421802 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:43.421807 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:43.421809 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:43.422061 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:43.422182 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:43.423407 (+ 1225us) server_negotiation.cc:299] Negotiation successful
0926 07:34:43.423527 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":326,"threads_started":1}
Sep 26 07:34:43 dist-test-slave-wdjc krb5kdc[12898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408083, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.252@KRBTEST.COM for kudu/127.22.53.253@KRBTEST.COM
I20260926 07:34:43.428366 12979 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39bf0f403cb7450eac133a20764b55c3: Bootstrap starting.
Sep 26 07:34:43 dist-test-slave-wdjc krb5kdc[12898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408083, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.253@KRBTEST.COM for kudu/127.22.53.252@KRBTEST.COM
I20260926 07:34:43.431409 12979 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 39bf0f403cb7450eac133a20764b55c3: Neither blocks nor log segments found. Creating new log.
I20260926 07:34:43.431445 13060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:43.419238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53120 (local address 127.22.53.253:46247)
0926 07:34:43.419377 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:43.419382 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:43.419395 (+ 13us) server_negotiation.cc:407] Connection header received
0926 07:34:43.419519 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:43.419522 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:43.419572 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:43.419652 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:43.420287 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:43.421217 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:43.423274 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:43.423635 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:43.423724 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:43.426958 (+ 3234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:43.426963 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:43.426966 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:43.427002 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:43.429200 (+ 2198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:43.429789 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:43.429793 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:43.429795 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:43.429893 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:43.430225 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:43.430229 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:43.430231 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:43.430580 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:43.430668 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:43.431068 (+ 400us) server_negotiation.cc:299] Negotiation successful
0926 07:34:43.431222 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 07:34:43.431963 13135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:43.419125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.253:46247 (local address 127.0.0.1:53120)
0926 07:34:43.419325 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:43.419338 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:43.419418 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:43.419788 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:43.419790 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:43.419983 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:43.420133 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:43.420142 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:43.421390 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:43.421394 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:43.423128 (+ 1734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:43.423138 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:43.423886 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:43.423889 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:43.423992 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:43.424749 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:43.424763 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:43.426775 (+ 2012us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:34:43.429358 (+ 2583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:43.429362 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:43.429364 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:43.429648 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:43.430004 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:43.430007 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:43.430011 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:43.430106 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:43.430750 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:34:43.430753 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:34:43.430866 (+ 113us) client_negotiation.cc:769] Sending connection context
0926 07:34:43.431635 (+ 769us) client_negotiation.cc:240] Negotiation successful
0926 07:34:43.431786 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118}
I20260926 07:34:43.432451 12979 log.cc:824] T 00000000000000000000000000000000 P 39bf0f403cb7450eac133a20764b55c3: Log is configured to *not* fsync() on all Append() calls
I20260926 07:34:43.432734 13131 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.53.252" port: 33181 } has no permanent_uuid. Determining permanent_uuid...
I20260926 07:34:43.433164 13057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:43.422234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.252:33181 (local address 127.0.0.1:36590)
0926 07:34:43.422417 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:43.422431 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:43.422509 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:43.422784 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:43.422787 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:43.422987 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:43.423095 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:43.423103 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:43.424410 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:43.424413 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:43.425292 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:43.425308 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:43.426242 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:43.426246 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:43.426336 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:43.427348 (+ 1012us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:43.427362 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:43.428975 (+ 1613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:34:43.431401 (+ 2426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:43.431405 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:43.431407 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:43.431794 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:43.432102 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:43.432105 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:43.432107 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:43.432200 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:43.432712 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:34:43.432718 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:34:43.432829 (+ 111us) client_negotiation.cc:769] Sending connection context
0926 07:34:43.432921 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 07:34:43.433057 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113}
I20260926 07:34:43.433733 13134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:43.422271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36590 (local address 127.22.53.252:33181)
0926 07:34:43.422381 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:43.422386 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:43.422478 (+ 92us) server_negotiation.cc:407] Connection header received
0926 07:34:43.422596 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:43.422602 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:43.422646 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:43.423203 (+ 557us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:43.423309 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:43.424282 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:43.425524 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:43.426001 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:43.426213 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:43.429193 (+ 2980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:43.429197 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:43.429201 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:43.429236 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:43.431245 (+ 2009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:43.431897 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:43.431901 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:43.431903 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:43.431981 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:43.432315 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:43.432318 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:43.432320 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:43.432606 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:43.433369 (+ 763us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:43.433441 (+ 72us) server_negotiation.cc:299] Negotiation successful
0926 07:34:43.433565 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
Sep 26 07:34:43 dist-test-slave-wdjc krb5kdc[12898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408082, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.53.254@KRBTEST.COM
I20260926 07:34:43.435360 12979 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39bf0f403cb7450eac133a20764b55c3: No bootstrap required, opened a new log
I20260926 07:34:43.437325 13054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07d8da00a0164cbe87f7570ab2a0361d: Bootstrap starting.
Sep 26 07:34:43 dist-test-slave-wdjc krb5kdc[12898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408083, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.53.252@KRBTEST.COM for kudu/127.22.53.252@KRBTEST.COM
I20260926 07:34:43.439797 12982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:43.420152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59296 (local address 127.22.53.254:34245)
0926 07:34:43.420277 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:43.420281 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:43.420335 (+ 54us) server_negotiation.cc:407] Connection header received
0926 07:34:43.420473 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:43.420476 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:43.420559 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:43.420644 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:43.421466 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:43.422148 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:43.423427 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:43.423748 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:43.423842 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:43.435886 (+ 12044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:43.435891 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:43.435894 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:43.435924 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:43.437769 (+ 1845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:43.438321 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:43.438327 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:43.438330 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:43.438416 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:43.438721 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:43.438725 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:43.438727 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:43.438957 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:43.439144 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:43.439508 (+ 364us) server_negotiation.cc:299] Negotiation successful
0926 07:34:43.439643 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 07:34:43.440423 13054 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 07d8da00a0164cbe87f7570ab2a0361d: Neither blocks nor log segments found. Creating new log.
I20260926 07:34:43.441547 12979 raft_consensus.cc:348] T 00000000000000000000000000000000 P 39bf0f403cb7450eac133a20764b55c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39bf0f403cb7450eac133a20764b55c3" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 34245 } } peers { permanent_uuid: "07d8da00a0164cbe87f7570ab2a0361d" member_type: VOTER last_known_addr { host: "127.22.53.253" port: 46247 } } peers { permanent_uuid: "cc65889b2a1145d3bad180231dc13a85" member_type: VOTER last_known_addr { host: "127.22.53.252" port: 33181 } }
I20260926 07:34:43.441857 12979 raft_consensus.cc:729] T 00000000000000000000000000000000 P 39bf0f403cb7450eac133a20764b55c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39bf0f403cb7450eac133a20764b55c3, State: Initialized, Role: FOLLOWER
I20260926 07:34:43.442021 13054 log.cc:824] T 00000000000000000000000000000000 P 07d8da00a0164cbe87f7570ab2a0361d: Log is configured to *not* fsync() on all Append() calls
I20260926 07:34:43.442274 12979 consensus_queue.cc:260] T 00000000000000000000000000000000 P 39bf0f403cb7450eac133a20764b55c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39bf0f403cb7450eac133a20764b55c3" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 34245 } } peers { permanent_uuid: "07d8da00a0164cbe87f7570ab2a0361d" member_type: VOTER last_known_addr { host: "127.22.53.253" port: 46247 } } peers { permanent_uuid: "cc65889b2a1145d3bad180231dc13a85" member_type: VOTER last_known_addr { host: "127.22.53.252" port: 33181 } }
I20260926 07:34:43.443178 13135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:43.433279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.53.252:33181 (local address 127.0.0.1:36600)
0926 07:34:43.433542 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0926 07:34:43.433557 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:34:43.433625 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:34:43.434179 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:34:43.434183 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:34:43.434352 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:34:43.434451 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:43.434458 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:43.435370 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:43.435373 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:43.436176 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:34:43.436187 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:34:43.436882 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:34:43.436885 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:34:43.437003 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:43.437402 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:34:43.437417 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:34:43.439005 (+ 1588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:34:43.441528 (+ 2523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:43.441531 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:43.441533 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:43.441771 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:43.442081 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:34:43.442083 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:34:43.442085 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:34:43.442168 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:34:43.442614 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:34:43.442620 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:34:43.442714 (+ 94us) client_negotiation.cc:769] Sending connection context
0926 07:34:43.442827 (+ 113us) client_negotiation.cc:240] Negotiation successful
0926 07:34:43.442978 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184}
I20260926 07:34:43.443603 13134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:43.433482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36600 (local address 127.22.53.252:33181)
0926 07:34:43.433927 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:43.433931 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:43.433943 (+ 12us) server_negotiation.cc:407] Connection header received
0926 07:34:43.433975 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:43.433977 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:43.434016 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:43.434088 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:43.434556 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:43.435255 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:43.436298 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:43.436666 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:43.436766 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:43.439141 (+ 2375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:43.439144 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:43.439147 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:43.439168 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:43.441409 (+ 2241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:43.441888 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:43.441891 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:43.441892 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:43.441986 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:43.442257 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:43.442259 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:43.442261 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:43.442486 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:43.442579 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:43.443371 (+ 792us) server_negotiation.cc:299] Negotiation successful
0926 07:34:43.443480 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385}
I20260926 07:34:43.443878 12979 sys_catalog.cc:564] T 00000000000000000000000000000000 P 39bf0f403cb7450eac133a20764b55c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:34:43.445061 13054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07d8da00a0164cbe87f7570ab2a0361d: No bootstrap required, opened a new log
I20260926 07:34:43.446905 13131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc65889b2a1145d3bad180231dc13a85: Bootstrap starting.
Sep 26 07:34:43 dist-test-slave-wdjc krb5kdc[12898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790408082, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.53.253@KRBTEST.COM
I20260926 07:34:43.450033 13131 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cc65889b2a1145d3bad180231dc13a85: Neither blocks nor log segments found. Creating new log.
I20260926 07:34:43.449671 13139 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39bf0f403cb7450eac133a20764b55c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39bf0f403cb7450eac133a20764b55c3" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 34245 } } peers { permanent_uuid: "07d8da00a0164cbe87f7570ab2a0361d" member_type: VOTER last_known_addr { host: "127.22.53.253" port: 46247 } } peers { permanent_uuid: "cc65889b2a1145d3bad180231dc13a85" member_type: VOTER last_known_addr { host: "127.22.53.252" port: 33181 } } }
I20260926 07:34:43.450169 13139 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39bf0f403cb7450eac133a20764b55c3 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:34:43.451030 13131 log.cc:824] T 00000000000000000000000000000000 P cc65889b2a1145d3bad180231dc13a85: Log is configured to *not* fsync() on all Append() calls
W20260926 07:34:43.452188 13150 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 39bf0f403cb7450eac133a20764b55c3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:34:43.452289 13150 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:34:43.452903 13060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:43.441323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53126 (local address 127.22.53.253:46247)
0926 07:34:43.441436 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:43.441441 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:43.441602 (+ 161us) server_negotiation.cc:407] Connection header received
0926 07:34:43.441788 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:43.441792 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:43.441848 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:43.441926 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:43.442848 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:43.443559 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:43.445377 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:43.445773 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:43.445864 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:43.449130 (+ 3266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:43.449135 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:43.449138 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:43.449176 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:43.450730 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:43.451257 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:43.451261 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:43.451263 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:43.451359 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:43.451682 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:43.451686 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:43.451688 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:43.451941 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:43.452131 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:43.452596 (+ 465us) server_negotiation.cc:299] Negotiation successful
0926 07:34:43.452747 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 07:34:43.454555 13131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc65889b2a1145d3bad180231dc13a85: No bootstrap required, opened a new log
I20260926 07:34:43.452389 12911 master_runner.cc:186] Error getting master registration for 127.22.53.253:46247: OK, instance_id { permanent_uuid: "07d8da00a0164cbe87f7570ab2a0361d" instance_seqno: 1790408083159213 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 07:34:43.455708 12911 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 07:34:43.456887 13054 raft_consensus.cc:348] T 00000000000000000000000000000000 P 07d8da00a0164cbe87f7570ab2a0361d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39bf0f403cb7450eac133a20764b55c3" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 34245 } } peers { permanent_uuid: "07d8da00a0164cbe87f7570ab2a0361d" member_type: VOTER last_known_addr { host: "127.22.53.253" port: 46247 } } peers { permanent_uuid: "cc65889b2a1145d3bad180231dc13a85" member_type: VOTER last_known_addr { host: "127.22.53.252" port: 33181 } }
I20260926 07:34:43.457242 13054 raft_consensus.cc:729] T 00000000000000000000000000000000 P 07d8da00a0164cbe87f7570ab2a0361d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07d8da00a0164cbe87f7570ab2a0361d, State: Initialized, Role: FOLLOWER
I20260926 07:34:43.457654 13054 consensus_queue.cc:260] T 00000000000000000000000000000000 P 07d8da00a0164cbe87f7570ab2a0361d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39bf0f403cb7450eac133a20764b55c3" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 34245 } } peers { permanent_uuid: "07d8da00a0164cbe87f7570ab2a0361d" member_type: VOTER last_known_addr { host: "127.22.53.253" port: 46247 } } peers { permanent_uuid: "cc65889b2a1145d3bad180231dc13a85" member_type: VOTER last_known_addr { host: "127.22.53.252" port: 33181 } }
I20260926 07:34:43.458715 13131 raft_consensus.cc:348] T 00000000000000000000000000000000 P cc65889b2a1145d3bad180231dc13a85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39bf0f403cb7450eac133a20764b55c3" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 34245 } } peers { permanent_uuid: "07d8da00a0164cbe87f7570ab2a0361d" member_type: VOTER last_known_addr { host: "127.22.53.253" port: 46247 } } peers { permanent_uuid: "cc65889b2a1145d3bad180231dc13a85" member_type: VOTER last_known_addr { host: "127.22.53.252" port: 33181 } }
I20260926 07:34:43.458976 13131 raft_consensus.cc:729] T 00000000000000000000000000000000 P cc65889b2a1145d3bad180231dc13a85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc65889b2a1145d3bad180231dc13a85, State: Initialized, Role: FOLLOWER
I20260926 07:34:43.459303 13131 consensus_queue.cc:260] T 00000000000000000000000000000000 P cc65889b2a1145d3bad180231dc13a85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39bf0f403cb7450eac133a20764b55c3" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 34245 } } peers { permanent_uuid: "07d8da00a0164cbe87f7570ab2a0361d" member_type: VOTER last_known_addr { host: "127.22.53.253" port: 46247 } } peers { permanent_uuid: "cc65889b2a1145d3bad180231dc13a85" member_type: VOTER last_known_addr { host: "127.22.53.252" port: 33181 } }
I20260926 07:34:43.459537 22743 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:34:43.459832 13154 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc65889b2a1145d3bad180231dc13a85 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39bf0f403cb7450eac133a20764b55c3" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 34245 } } peers { permanent_uuid: "07d8da00a0164cbe87f7570ab2a0361d" member_type: VOTER last_known_addr { host: "127.22.53.253" port: 46247 } } peers { permanent_uuid: "cc65889b2a1145d3bad180231dc13a85" member_type: VOTER last_known_addr { host: "127.22.53.252" port: 33181 } } }
I20260926 07:34:43.459971 13154 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc65889b2a1145d3bad180231dc13a85 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:34:43.460125 13054 sys_catalog.cc:564] T 00000000000000000000000000000000 P 07d8da00a0164cbe87f7570ab2a0361d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:34:43.460644 13131 sys_catalog.cc:564] T 00000000000000000000000000000000 P cc65889b2a1145d3bad180231dc13a85 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:34:43.460853 12953 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "39bf0f403cb7450eac133a20764b55c3"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59296
I20260926 07:34:43.461130 12953 raft_consensus.cc:482] T 00000000000000000000000000000000 P 39bf0f403cb7450eac133a20764b55c3 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 07:34:43.461292 12953 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 39bf0f403cb7450eac133a20764b55c3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:34:43.462805 12953 raft_consensus.cc:504] T 00000000000000000000000000000000 P 39bf0f403cb7450eac133a20764b55c3 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39bf0f403cb7450eac133a20764b55c3" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 34245 } } peers { permanent_uuid: "07d8da00a0164cbe87f7570ab2a0361d" member_type: VOTER last_known_addr { host: "127.22.53.253" port: 46247 } } peers { permanent_uuid: "cc65889b2a1145d3bad180231dc13a85" member_type: VOTER last_known_addr { host: "127.22.53.252" port: 33181 } }
I20260926 07:34:43.464752 12953 leader_election.cc:288] T 00000000000000000000000000000000 P 39bf0f403cb7450eac133a20764b55c3 [CANDIDATE]: Term 1 election: Requested vote from peers 07d8da00a0164cbe87f7570ab2a0361d (127.22.53.253:46247), cc65889b2a1145d3bad180231dc13a85 (127.22.53.252:33181)
I20260926 07:34:43.465071 13028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "39bf0f403cb7450eac133a20764b55c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "07d8da00a0164cbe87f7570ab2a0361d"
I20260926 07:34:43.465337 13028 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 07d8da00a0164cbe87f7570ab2a0361d [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:34:43.466694 13028 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 07d8da00a0164cbe87f7570ab2a0361d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39bf0f403cb7450eac133a20764b55c3 in term 1.
I20260926 07:34:43.468029 12930 leader_election.cc:302] T 00000000000000000000000000000000 P 39bf0f403cb7450eac133a20764b55c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07d8da00a0164cbe87f7570ab2a0361d, 39bf0f403cb7450eac133a20764b55c3; no voters:
I20260926 07:34:43.468380 13139 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 39bf0f403cb7450eac133a20764b55c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:34:43.468669 13139 raft_consensus.cc:686] T 00000000000000000000000000000000 P 39bf0f403cb7450eac133a20764b55c3 [term 1 LEADER]: Becoming Leader. State: Replica: 39bf0f403cb7450eac133a20764b55c3, State: Running, Role: LEADER
I20260926 07:34:43.469063 13139 consensus_queue.cc:237] T 00000000000000000000000000000000 P 39bf0f403cb7450eac133a20764b55c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39bf0f403cb7450eac133a20764b55c3" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 34245 } } peers { permanent_uuid: "07d8da00a0164cbe87f7570ab2a0361d" member_type: VOTER last_known_addr { host: "127.22.53.253" port: 46247 } } peers { permanent_uuid: "cc65889b2a1145d3bad180231dc13a85" member_type: VOTER last_known_addr { host: "127.22.53.252" port: 33181 } }
I20260926 07:34:43.469028 13105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "39bf0f403cb7450eac133a20764b55c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "cc65889b2a1145d3bad180231dc13a85"
I20260926 07:34:43.469308 13105 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cc65889b2a1145d3bad180231dc13a85 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:34:43.470950 13105 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cc65889b2a1145d3bad180231dc13a85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39bf0f403cb7450eac133a20764b55c3 in term 1.
I20260926 07:34:43.471158 13139 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39bf0f403cb7450eac133a20764b55c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 39bf0f403cb7450eac133a20764b55c3. Latest consensus state: current_term: 1 leader_uuid: "39bf0f403cb7450eac133a20764b55c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39bf0f403cb7450eac133a20764b55c3" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 34245 } } peers { permanent_uuid: "07d8da00a0164cbe87f7570ab2a0361d" member_type: VOTER last_known_addr { host: "127.22.53.253" port: 46247 } } peers { permanent_uuid: "cc65889b2a1145d3bad180231dc13a85" member_type: VOTER last_known_addr { host: "127.22.53.252" port: 33181 } } }
I20260926 07:34:43.471287 13139 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39bf0f403cb7450eac133a20764b55c3 [sys.catalog]: This master's current role is: LEADER
I20260926 07:34:43.475863 12986 master_runner.cc:263] No leader master found from master 07d8da00a0164cbe87f7570ab2a0361d
I20260926 07:34:43.475955 12986 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 07:34:43.479455 13153 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07d8da00a0164cbe87f7570ab2a0361d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39bf0f403cb7450eac133a20764b55c3" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 34245 } } peers { permanent_uuid: "07d8da00a0164cbe87f7570ab2a0361d" member_type: VOTER last_known_addr { host: "127.22.53.253" port: 46247 } } peers { permanent_uuid: "cc65889b2a1145d3bad180231dc13a85" member_type: VOTER last_known_addr { host: "127.22.53.252" port: 33181 } } }
I20260926 07:34:43.479620 13153 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07d8da00a0164cbe87f7570ab2a0361d [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:34:43.480041 13134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:43.467316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36612 (local address 127.22.53.252:33181)
0926 07:34:43.468646 (+ 1330us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:43.468651 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:43.468667 (+ 16us) server_negotiation.cc:407] Connection header received
0926 07:34:43.468715 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:43.468721 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:43.468780 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:43.468864 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:43.471068 (+ 2204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:43.471838 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:43.473067 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:43.473397 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:43.473476 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:43.475714 (+ 2238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:43.475718 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:43.475721 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:43.475748 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:43.477973 (+ 2225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:43.478504 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:43.478508 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:43.478510 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:43.478615 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:43.478961 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:43.478968 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:43.478972 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:43.479220 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:43.479331 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:43.479740 (+ 409us) server_negotiation.cc:299] Negotiation successful
0926 07:34:43.479882 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1247}
I20260926 07:34:43.480762 12982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:43.467071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59304 (local address 127.22.53.254:34245)
0926 07:34:43.467202 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:43.467207 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:43.469120 (+ 1913us) server_negotiation.cc:407] Connection header received
0926 07:34:43.469420 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:43.469423 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:43.469481 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:43.469533 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:43.470628 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:43.471394 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:43.472645 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:43.472968 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:43.473047 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:43.475080 (+ 2033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:43.475084 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:43.475087 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:43.475118 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:43.476871 (+ 1753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:43.478924 (+ 2053us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:43.478927 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:43.478929 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:43.479032 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:43.479350 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:43.479353 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:43.479355 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:43.479575 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:43.479688 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:43.480425 (+ 737us) server_negotiation.cc:299] Negotiation successful
0926 07:34:43.480596 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 07:34:43.480903 13186 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:34:43.481999 13186 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260926 07:34:43.484843 13176 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 07d8da00a0164cbe87f7570ab2a0361d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:34:43.484956 13176 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260926 07:34:43.488981 13175 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cc65889b2a1145d3bad180231dc13a85: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:34:43.489099 13175 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:34:43.490442 13060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:43.467884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53138 (local address 127.22.53.253:46247)
0926 07:34:43.467998 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:43.468002 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:43.480966 (+ 12964us) server_negotiation.cc:407] Connection header received
0926 07:34:43.481151 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:43.481154 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:43.481214 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:43.481275 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:43.482206 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:43.483011 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:43.484173 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:43.484519 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:43.484622 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:43.486574 (+ 1952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:34:43.486579 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:34:43.486582 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:34:43.486615 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:34:43.488375 (+ 1760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:43.488987 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:43.488991 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:43.488993 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:43.489083 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:34:43.489390 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:34:43.489394 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:34:43.489395 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:34:43.489667 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:34:43.489819 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:43.490134 (+ 315us) server_negotiation.cc:299] Negotiation successful
0926 07:34:43.490270 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 07:34:43.493399 13105 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cc65889b2a1145d3bad180231dc13a85 [term 1 FOLLOWER]: Refusing update from remote peer 39bf0f403cb7450eac133a20764b55c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:34:43.494570 13183 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 39bf0f403cb7450eac133a20764b55c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc65889b2a1145d3bad180231dc13a85" member_type: VOTER last_known_addr { host: "127.22.53.252" port: 33181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:34:43.499733 13154 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc65889b2a1145d3bad180231dc13a85 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 39bf0f403cb7450eac133a20764b55c3. Latest consensus state: current_term: 1 leader_uuid: "39bf0f403cb7450eac133a20764b55c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39bf0f403cb7450eac133a20764b55c3" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 34245 } } peers { permanent_uuid: "07d8da00a0164cbe87f7570ab2a0361d" member_type: VOTER last_known_addr { host: "127.22.53.253" port: 46247 } } peers { permanent_uuid: "cc65889b2a1145d3bad180231dc13a85" member_type: VOTER last_known_addr { host: "127.22.53.252" port: 33181 } } }
I20260926 07:34:43.499914 13154 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc65889b2a1145d3bad180231dc13a85 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:34:43.501626 13028 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 07d8da00a0164cbe87f7570ab2a0361d [term 1 FOLLOWER]: Refusing update from remote peer 39bf0f403cb7450eac133a20764b55c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:34:43.502305 13183 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 39bf0f403cb7450eac133a20764b55c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07d8da00a0164cbe87f7570ab2a0361d" member_type: VOTER last_known_addr { host: "127.22.53.253" port: 46247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:34:43.506613 13153 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07d8da00a0164cbe87f7570ab2a0361d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 39bf0f403cb7450eac133a20764b55c3. Latest consensus state: current_term: 1 leader_uuid: "39bf0f403cb7450eac133a20764b55c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39bf0f403cb7450eac133a20764b55c3" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 34245 } } peers { permanent_uuid: "07d8da00a0164cbe87f7570ab2a0361d" member_type: VOTER last_known_addr { host: "127.22.53.253" port: 46247 } } peers { permanent_uuid: "cc65889b2a1145d3bad180231dc13a85" member_type: VOTER last_known_addr { host: "127.22.53.252" port: 33181 } } }
I20260926 07:34:43.506776 13153 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07d8da00a0164cbe87f7570ab2a0361d [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:34:43.507527 13139 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39bf0f403cb7450eac133a20764b55c3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "39bf0f403cb7450eac133a20764b55c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39bf0f403cb7450eac133a20764b55c3" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 34245 } } peers { permanent_uuid: "07d8da00a0164cbe87f7570ab2a0361d" member_type: VOTER last_known_addr { host: "127.22.53.253" port: 46247 } } peers { permanent_uuid: "cc65889b2a1145d3bad180231dc13a85" member_type: VOTER last_known_addr { host: "127.22.53.252" port: 33181 } } }
I20260926 07:34:43.507676 13139 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39bf0f403cb7450eac133a20764b55c3 [sys.catalog]: This master's current role is: LEADER
I20260926 07:34:43.508000 13188 mvcc.cc:205] Tried to move back new op lower bound from 7333511509980356608 to 7333511509893341184. Current Snapshot: MvccSnapshot[applied={T|T < 7333511509980356608}]
I20260926 07:34:43.508139 13154 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc65889b2a1145d3bad180231dc13a85 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "39bf0f403cb7450eac133a20764b55c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39bf0f403cb7450eac133a20764b55c3" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 34245 } } peers { permanent_uuid: "07d8da00a0164cbe87f7570ab2a0361d" member_type: VOTER last_known_addr { host: "127.22.53.253" port: 46247 } } peers { permanent_uuid: "cc65889b2a1145d3bad180231dc13a85" member_type: VOTER last_known_addr { host: "127.22.53.252" port: 33181 } } }
I20260926 07:34:43.508289 13154 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc65889b2a1145d3bad180231dc13a85 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:34:43.509406 13183 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39bf0f403cb7450eac133a20764b55c3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "39bf0f403cb7450eac133a20764b55c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39bf0f403cb7450eac133a20764b55c3" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 34245 } } peers { permanent_uuid: "07d8da00a0164cbe87f7570ab2a0361d" member_type: VOTER last_known_addr { host: "127.22.53.253" port: 46247 } } peers { permanent_uuid: "cc65889b2a1145d3bad180231dc13a85" member_type: VOTER last_known_addr { host: "127.22.53.252" port: 33181 } } }
I20260926 07:34:43.509546 13183 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39bf0f403cb7450eac133a20764b55c3 [sys.catalog]: This master's current role is: LEADER
I20260926 07:34:43.510639 13153 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07d8da00a0164cbe87f7570ab2a0361d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "39bf0f403cb7450eac133a20764b55c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39bf0f403cb7450eac133a20764b55c3" member_type: VOTER last_known_addr { host: "127.22.53.254" port: 34245 } } peers { permanent_uuid: "07d8da00a0164cbe87f7570ab2a0361d" member_type: VOTER last_known_addr { host: "127.22.53.253" port: 46247 } } peers { permanent_uuid: "cc65889b2a1145d3bad180231dc13a85" member_type: VOTER last_known_addr { host: "127.22.53.252" port: 33181 } } }
I20260926 07:34:43.510772 13153 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07d8da00a0164cbe87f7570ab2a0361d [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:34:43.511188 13190 mvcc.cc:205] Tried to move back new op lower bound from 7333511509980356608 to 7333511509893341184. Current Snapshot: MvccSnapshot[applied={T|T < 7333511509980356608}]
I20260926 07:34:43.513741 13192 mvcc.cc:205] Tried to move back new op lower bound from 7333511509980356608 to 7333511509893341184. Current Snapshot: MvccSnapshot[applied={T|T < 7333511509980356608}]
I20260926 07:34:43.517558 13186 catalog_manager.cc:1347] Generated new cluster ID: 5579d36a7fea441b981a2b911e566bf1
I20260926 07:34:43.517647 13186 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:34:43.540222 13186 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:34:43.541639 13186 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:34:43.564426 13186 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 39bf0f403cb7450eac133a20764b55c3: Generated new TSK 0
I20260926 07:34:43.565585 13186 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 07:34:43.610988 12982 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:34:43.606557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59308 (local address 127.22.53.254:34245)
0926 07:34:43.606710 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:43.606715 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:43.609387 (+ 2672us) server_negotiation.cc:407] Connection header received
0926 07:34:43.609889 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:43.609893 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:43.609959 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:43.610023 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:43.610815 (+ 792us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59308: BlockingRecv error: recv got EOF from 127.0.0.1:59308 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260926 07:34:43.612417 13060 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:34:43.607416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53144 (local address 127.22.53.253:46247)
0926 07:34:43.607552 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:43.607557 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:43.609780 (+ 2223us) server_negotiation.cc:407] Connection header received
0926 07:34:43.611456 (+ 1676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:43.611461 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:43.611524 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:43.611586 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:43.612276 (+ 690us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53144: BlockingRecv error: recv got EOF from 127.0.0.1:53144 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260926 07:34:43.614171 13134 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:34:43.608746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36614 (local address 127.22.53.252:33181)
0926 07:34:43.608897 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:43.608902 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:43.613078 (+ 4176us) server_negotiation.cc:407] Connection header received
0926 07:34:43.613214 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:43.613218 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:43.613279 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:43.613382 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:43.614027 (+ 645us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36614: BlockingRecv error: recv got EOF from 127.0.0.1:36614 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260926 07:34:43.619891 13060 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:34:43.617998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53150 (local address 127.22.53.253:46247)
0926 07:34:43.618131 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:43.618137 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:43.618691 (+ 554us) server_negotiation.cc:407] Connection header received
0926 07:34:43.618970 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:43.618974 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:43.619065 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:43.619128 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:43.619755 (+ 627us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53150: BlockingRecv error: recv got EOF from 127.0.0.1:53150 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260926 07:34:43.626008 13060 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:34:43.624045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53166 (local address 127.22.53.253:46247)
0926 07:34:43.624180 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:43.624185 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:43.624855 (+ 670us) server_negotiation.cc:407] Connection header received
0926 07:34:43.625082 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:43.625086 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:43.625148 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:43.625208 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:43.625872 (+ 664us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53166: BlockingRecv error: recv got EOF from 127.0.0.1:53166 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260926 07:34:43.632268 13060 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:34:43.630529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53176 (local address 127.22.53.253:46247)
0926 07:34:43.630653 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:43.630658 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:43.631238 (+ 580us) server_negotiation.cc:407] Connection header received
0926 07:34:43.631446 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:43.631450 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:43.631504 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:43.631557 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:43.632140 (+ 583us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53176: BlockingRecv error: recv got EOF from 127.0.0.1:53176 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260926 07:34:43.645210 13060 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:34:43.640152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53186 (local address 127.22.53.253:46247)
0926 07:34:43.640320 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:43.640325 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:43.643879 (+ 3554us) server_negotiation.cc:407] Connection header received
0926 07:34:43.644111 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:43.644115 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:43.644176 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:43.644236 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:43.645074 (+ 838us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53186: BlockingRecv error: recv got EOF from 127.0.0.1:53186 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
W20260926 07:34:43.659132 13060 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:34:43.657090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53188 (local address 127.22.53.253:46247)
0926 07:34:43.657225 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:43.657230 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:43.657983 (+ 753us) server_negotiation.cc:407] Connection header received
0926 07:34:43.658215 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:43.658219 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:43.658279 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:43.658337 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:43.658986 (+ 649us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53188: BlockingRecv error: recv got EOF from 127.0.0.1:53188 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260926 07:34:43.681459 13060 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:34:43.678514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53192 (local address 127.22.53.253:46247)
0926 07:34:43.678659 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:43.678663 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:43.679327 (+ 664us) server_negotiation.cc:407] Connection header received
0926 07:34:43.679585 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:43.679588 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:43.679649 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:43.679711 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:43.681329 (+ 1618us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53192: BlockingRecv error: recv got EOF from 127.0.0.1:53192 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260926 07:34:43.717841 13060 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:34:43.716468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53208 (local address 127.22.53.253:46247)
0926 07:34:43.716654 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:43.716660 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:43.716945 (+ 285us) server_negotiation.cc:407] Connection header received
0926 07:34:43.717012 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:43.717015 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:43.717066 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:43.717145 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:43.717711 (+ 566us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53208: BlockingRecv error: recv got EOF from 127.0.0.1:53208 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
W20260926 07:34:43.785286 13060 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:34:43.784251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53210 (local address 127.22.53.253:46247)
0926 07:34:43.784412 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:43.784416 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:43.784432 (+ 16us) server_negotiation.cc:407] Connection header received
0926 07:34:43.784490 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:43.784493 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:43.784566 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:43.784666 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:43.785180 (+ 514us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53210: BlockingRecv error: recv got EOF from 127.0.0.1:53210 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260926 07:34:43.917135 13060 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:34:43.915894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53216 (local address 127.22.53.253:46247)
0926 07:34:43.916048 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:43.916052 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:43.916176 (+ 124us) server_negotiation.cc:407] Connection header received
0926 07:34:43.916319 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:43.916323 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:43.916372 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:43.916458 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:43.916993 (+ 535us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53216: BlockingRecv error: recv got EOF from 127.0.0.1:53216 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260926 07:34:44.176786 13060 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:34:44.175549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53218 (local address 127.22.53.253:46247)
0926 07:34:44.175690 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:44.175697 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:44.175804 (+ 107us) server_negotiation.cc:407] Connection header received
0926 07:34:44.175982 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:44.175986 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:44.176042 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:44.176139 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:34:44.176649 (+ 510us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53218: BlockingRecv error: recv got EOF from 127.0.0.1:53218 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 07:34:44.486413 13176 catalog_manager.cc:1259] Loaded cluster ID: 5579d36a7fea441b981a2b911e566bf1
I20260926 07:34:44.486523 13176 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 07d8da00a0164cbe87f7570ab2a0361d: loading cluster ID for follower catalog manager: success
I20260926 07:34:44.488669 13176 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 07d8da00a0164cbe87f7570ab2a0361d: acquiring CA information for follower catalog manager: success
I20260926 07:34:44.490352 13176 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 07d8da00a0164cbe87f7570ab2a0361d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 07:34:44.490487 13175 catalog_manager.cc:1259] Loaded cluster ID: 5579d36a7fea441b981a2b911e566bf1
I20260926 07:34:44.490537 13175 catalog_manager.cc:1552] T 00000000000000000000000000000000 P cc65889b2a1145d3bad180231dc13a85: loading cluster ID for follower catalog manager: success
I20260926 07:34:44.492365 13175 catalog_manager.cc:1574] T 00000000000000000000000000000000 P cc65889b2a1145d3bad180231dc13a85: acquiring CA information for follower catalog manager: success
I20260926 07:34:44.493593 13175 catalog_manager.cc:1602] T 00000000000000000000000000000000 P cc65889b2a1145d3bad180231dc13a85: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 07:34:44.695294 13262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:44.691303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53230 (local address 127.22.53.253:46247)
0926 07:34:44.691641 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:44.691645 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:44.691661 (+ 16us) server_negotiation.cc:407] Connection header received
0926 07:34:44.691774 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:44.691776 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:44.691908 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:44.692011 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:34:44.692322 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:44.692920 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:44.693799 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:44.694019 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:44.694096 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:44.694396 (+ 300us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:34:44.694875 (+ 479us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:34:44.694968 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:44.695038 (+ 70us) server_negotiation.cc:299] Negotiation successful
0926 07:34:44.695091 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":103,"threads_started":1}
I20260926 07:34:44.702514 13268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:44.698830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36626 (local address 127.22.53.252:33181)
0926 07:34:44.699063 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:44.699066 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:44.699079 (+ 13us) server_negotiation.cc:407] Connection header received
0926 07:34:44.699172 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:44.699175 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:44.699297 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:44.699408 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:34:44.699620 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:44.700132 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:44.701054 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:44.701285 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:44.701387 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:44.701631 (+ 244us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:34:44.702095 (+ 464us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:34:44.702187 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:44.702247 (+ 60us) server_negotiation.cc:299] Negotiation successful
0926 07:34:44.702303 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":79,"threads_started":1}
I20260926 07:34:44.710871 13268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:44.706620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36630 (local address 127.22.53.252:33181)
0926 07:34:44.706798 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:44.706803 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:44.706817 (+ 14us) server_negotiation.cc:407] Connection header received
0926 07:34:44.707098 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:44.707105 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:44.707174 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:44.707228 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:34:44.707668 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:44.708281 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:44.709524 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:44.709872 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:44.709974 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:44.710326 (+ 352us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:34:44.710434 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:34:44.710488 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:44.710638 (+ 150us) server_negotiation.cc:299] Negotiation successful
0926 07:34:44.710701 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260926 07:34:44.710954 13262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:44.706750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53242 (local address 127.22.53.253:46247)
0926 07:34:44.706889 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:44.706893 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:44.707274 (+ 381us) server_negotiation.cc:407] Connection header received
0926 07:34:44.707369 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:44.707373 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:44.707431 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:44.707504 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:34:44.707816 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:44.708652 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:44.709803 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:44.710121 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:44.710224 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:44.710472 (+ 248us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:34:44.710560 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:34:44.710624 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:44.710770 (+ 146us) server_negotiation.cc:299] Negotiation successful
0926 07:34:44.710830 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 07:34:44.711608 13274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:34:44.706242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59310 (local address 127.22.53.254:34245)
0926 07:34:44.706562 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0926 07:34:44.706566 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:34:44.706866 (+ 300us) server_negotiation.cc:407] Connection header received
0926 07:34:44.707277 (+ 411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:34:44.707281 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:34:44.707464 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:34:44.707607 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:34:44.707894 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:44.708704 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:44.709638 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:34:44.709988 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:34:44.710088 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:34:44.710706 (+ 618us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:34:44.711117 (+ 411us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:34:44.711188 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 07:34:44.711377 (+ 189us) server_negotiation.cc:299] Negotiation successful
0926 07:34:44.711451 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":131,"threads_started":1}
I20260926 07:34:44.714996 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 12911
I20260926 07:34:44.722235 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 12986
I20260926 07:34:44.728513 22743 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdIE4X7/build/debug/bin/kudu with pid 13062
2026-09-26T07:34:44Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3186 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6354 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (301193 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST